From e521d64ca42c5314bb0ccddb6186db9aaa5e0532 Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Fri, 10 Jul 2026 22:09:30 +0200 Subject: [PATCH 01/53] Work out plan for asset pipeline --- Research/AssetPipeline.md | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) create mode 100644 Research/AssetPipeline.md diff --git a/Research/AssetPipeline.md b/Research/AssetPipeline.md new file mode 100644 index 0000000..5345d53 --- /dev/null +++ b/Research/AssetPipeline.md @@ -0,0 +1,27 @@ +# Asset Pipeline +Now that I have support for encoding textures offline and then loading/transcoding them when the game runs (see CapriKit.SuperCompressed) it is time to start work on a real asset pipeline. I want you to help me create a high-level architecture for it. The expected output (when we are ready for that) is a markdown file in the `Research` folder. + +# Goals +1. Encoding/compile/compress assets into formats that makes them easy to ship and easy load +2. Load the assets when requested by the game into ready-to-use objects (for example, when done a texture is uploaded to the GPU and ready to be referenced) + +## Subgoals +1. Flexible enough to support textures, shaders, models and audio in a similar process +2. Support for detecting changes on-disk and then rebuilding and hot-reloading them +3. Loading a groups of assets (bundles) in parallel while the main loop keeps running uninterrupted. + 3a. I am thinking of a process similar to the parallel loading of test screens in the background in `C:\projects\csharp\CapriKit\source\CapriKit.Tests.Tool\Program.cs` with a `LightweightChannel` `C:\projects\csharp\CapriKit\source\CapriKit.Concurrency\Primitives\LightweightChannel.cs` + 3b. I wonder if I need two steps here, full background work and then coordination with the main loop to finish the loading of things like textures that need to be uploaded to GPU memory, which I think you cannot do in parallel/without a DeviceContext and help of the main thread. + 3c. I need a way to identify assets (maybe just a record `Asset` or `Asset` which holds the relative path to the asset). If I load a bundle I should be able to wait for the bundle to be fully loaded and then get the concrete Texture, Model, etc... object from the bundle using that record. + 3d. Somehow that Texture, model, etc.. object needs to have a point of indirection to support that hot reloading + + +I am saying bundle here as a helpful abstraction in code, but on disk I want each file to stay independent. I know it is a common optimization to compress multiple assets into one file and load those together (better disk IO) but I think that will complicate hot reloading and other IO code. (Happy to be proven wrong though, so maybe discuss it as an extra option but assume for now we are not doing it). + +## Assumptions +1. Assume assets do not need to carry extra information for the content pipeline. For example, if a texture should be loaded as normal map, srgb or linear texture is determined by the caller at run-time, not via an extra config file. +2. Assume that if an asset is loaded it stays loaded, we're more going for games like Factorio, Stationeers, Satisfactory and not for level-based games. + +## Examples +I have created a similar system before. See `C:\projects\csharp\MiniEngine3\src\Mini.Engine.Content\` and especially `C:\projects\csharp\MiniEngine3\src\Mini.Engine.Content\ContentManager.cs` and `C:\projects\csharp\MiniEngine3\src\Mini.Engine.Content\ContentProcessor.cs` but there were a couple of problems +- Background loading was super hacky +- A lot of boilerplate code, for example in the folder `C:\projects\csharp\MiniEngine3\src\Mini.Engine.Content\Shaders\` you can see that I needed 3 classes for each type of shader. I really want to have to use a lot less code. From 77e156e40cb34a4a9e9ec932fb691173685d4f24 Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Sat, 11 Jul 2026 12:50:55 +0200 Subject: [PATCH 02/53] Early asset pipeline implementation --- CapriKit.slnx | 3 + Research/AssetPipelineArchitecture.md | 234 ++++++++++++++++++ .../CapriKit.AssetPipeline/AssetTranscoder.cs | 122 +++++++++ .../CapriKit.AssetPipeline.csproj | 5 + .../CapriKit.AssetPipeline/IAssetCompiler.cs | 34 +++ .../Assets/ITextureEncoder.cs | 10 + .../Assets/VertexShaderTranscoder.cs | 44 ++++ .../CapriKit.DirectX11.csproj | 1 + .../Resources/Shaders/VertexShader.cs | 12 +- .../Buffers/IBufferWriterExtensions.cs | 60 +++++ .../Buffers/SequenceReaderExtensions.cs | 101 ++++++++ source/CapriKit.IO/CapriKit.IO.csproj | 4 +- .../AssetPipeline/AssetTranscoderTests.cs | 82 ++++++ source/CapriKit.Tests/CapriKit.Tests.csproj | 1 + .../Buffers/IBufferWriterExtensionsTests.cs | 76 ++++++ .../Buffers/SequenceReaderExtensionsTests.cs | 109 ++++++++ 16 files changed, 891 insertions(+), 7 deletions(-) create mode 100644 Research/AssetPipelineArchitecture.md create mode 100644 source/CapriKit.AssetPipeline/AssetTranscoder.cs create mode 100644 source/CapriKit.AssetPipeline/CapriKit.AssetPipeline.csproj create mode 100644 source/CapriKit.AssetPipeline/IAssetCompiler.cs create mode 100644 source/CapriKit.DirectX11/Assets/ITextureEncoder.cs create mode 100644 source/CapriKit.DirectX11/Assets/VertexShaderTranscoder.cs create mode 100644 source/CapriKit.IO/Buffers/IBufferWriterExtensions.cs create mode 100644 source/CapriKit.IO/Buffers/SequenceReaderExtensions.cs create mode 100644 source/CapriKit.Tests/AssetPipeline/AssetTranscoderTests.cs create mode 100644 source/CapriKit.Tests/IO/Buffers/IBufferWriterExtensionsTests.cs create mode 100644 source/CapriKit.Tests/IO/Buffers/SequenceReaderExtensionsTests.cs diff --git a/CapriKit.slnx b/CapriKit.slnx index 818c7eb..9e27b76 100644 --- a/CapriKit.slnx +++ b/CapriKit.slnx @@ -32,6 +32,9 @@ + + + diff --git a/Research/AssetPipelineArchitecture.md b/Research/AssetPipelineArchitecture.md new file mode 100644 index 0000000..fd0ff6d --- /dev/null +++ b/Research/AssetPipelineArchitecture.md @@ -0,0 +1,234 @@ +# Asset Pipeline — High-Level Architecture + +Companion to `AssetPipeline.md` (the brief). Status: draft for discussion, 2026-07-10. + +## Decisions + +| Decision | Choice | Rationale | +|---|---|---| +| Build timing | **Hybrid** | Dev builds compile stale assets on demand into a cache; a CLI step produces the same output for shipping. Shipped games load precompiled assets only and carry no compiler code. | +| Asset relationships | **Flat** | No cross-asset references. Game code composes (loads a model's geometry and its textures separately). Kills the dependency-graph machinery that made MiniEngine3 complex. | +| GPU coupling | **Direct DX11 dependency** | The pipeline references `CapriKit.DirectX11` and produces GPU resources itself. Fewer abstractions to understand and maintain. | +| Shaders | **Offline bytecode** | HLSL → DXBC at compile time. `CapriKit.Generators.HLSL` keeps generating the C#-side metadata (structs, input element descriptions, entry points) unchanged; the pipeline only takes over bytecode compilation. | + +## Big Picture + +``` + source tree compiled tree runtime + (loose files) (loose files, mirrored) + ┌──────────────┐ compile ┌──────────────┐ load ┌───────────────┐ + │ grass.png │ ─────────► │ grass.cka │ ─────────► │ Texture2D │ + │ basic.hlsl │ IAsset- │ basic.cka │ IAsset- │ VertexShader..│ + │ tree.gltf │ Compiler │ tree.cka │ Loader │ Model │ + │ click.wav │ │ click.cka │ │ AudioClip │ + └──────────────┘ └──────────────┘ └───────────────┘ + dev machine only dev cache == ship content game process +``` + +Two phases, two families of pluggable pieces: + +- **Compilers** (dev machine / build server): source format → compiled container. Slow, thorough, offline. +- **Loaders** (game process): compiled container → ready-to-use object (texture on the GPU, playable audio clip). Fast, allocation-conscious. + +An asset is identified by its **source-relative path**, and every source file maps 1:1 to +one compiled file with the same relative path (different extension). This 1:1 rule is what +keeps identity, staleness checks, and file watching trivial. + +## Phase 1: Compiling + +```csharp +public interface IAssetCompiler +{ + IReadOnlySet SupportedExtensions { get; } + int Version { get; } + + // Reads source via a *tracking* file system so every file touched + // (e.g. #included HLSL) is recorded as an input of this asset. + void Compile(AssetId id, ITrackingReadOnlyFileSystem source, Stream output); +} +``` + +### Compiled container (`.cka` — "CapriKit Asset") + +One small binary envelope shared by all types, so staleness and loading logic is written once: + +- Header: magic, container version, compiler type + version, hashes of all input files. +- Body: one or more named blobs (a KTX2 payload; a DXBC blob per shader entry point; vertex + index buffers). + +### Staleness & the hybrid model + +An asset is stale when the compiled file is missing, or the stored input hashes / compiler +version no longer match. The check is identical everywhere; only *who runs it* differs: + +- **Dev**: the game process checks staleness when an asset is first requested and recompiles + inline (on the worker thread that is loading it) before loading. First run is slow, then it's cache hits. +- **Ship**: the CLI tool walks the source tree, compiles everything stale, and the resulting + compiled tree *is* the shipped content folder. Shipped builds skip the staleness check entirely + and never reference the compiler assemblies. + +### Per-type mapping + +| Type | Source | Compile step | Compiled body | +|---|---|---|---| +| Texture | png, jpg, ... | `CapriKit.SuperCompressed` encoder (mips baked in) | KTX2 | +| Shader | hlsl | DXC/FXC per `#pragma`-marked entry point | DXBC blob per entry point | +| Model | tbd (gltf?) | parse, triangulate, build interleaved buffers | vertex + index blobs | +| Audio | wav, ogg | tbd (likely near pass-through) | PCM or compressed blob | + +Note that input-file tracking (an `.hlsl` including `common.hlsli`) is a *build* dependency, +not a runtime asset reference — it exists only so staleness and hot reload know that touching +`common.hlsli` dirties every shader that included it. The "flat assets" decision is untouched. + +## Phase 2: Loading + +### Identity and typed access + +```csharp +public sealed record Asset(string Path); // relative path, forward slashes, lower case + +public static class GameAssets // game code declares what it uses +{ + public static readonly Asset Grass = new("textures/grass.png"); + // snip +} +``` + +Per assumption 1, interpretation (sRGB vs linear, transcode target, ...) is supplied by the +caller **at request time** as an optional per-type options record; nothing is persisted. + +### Bundles and background loading + +This generalizes the pattern already proven in `CapriKit.Tests.Tool/Program.cs`: +`BackgroundWorker` jobs write results (or one exception) into a `LightweightChannel`, +and the main loop drains it once per frame. + +```csharp +var bundle = assets.StartLoading([GameAssets.Grass, GameAssets.Tree /* snip */]); + +// in the main loop, once per frame: +assets.Update(); // drains channels, publishes finished assets, applies hot reloads + +if (bundle.IsLoaded) // all jobs finished (faulted bundles rethrow here) +{ + AssetRef grass = bundle.Get(GameAssets.Grass); +} +``` + +`bundle.Progress` (loaded/total) falls out for free for loading screens. + +### Threading: answering "do I need two steps?" (brief 3b) + +Mostly no, on D3D11. `ID3D11Device` is **free-threaded**: creating textures, buffers, and +shaders — including uploading initial data — is legal from any thread. Only the +**immediate `DeviceContext`** is single-threaded. Since mips are baked offline, nothing in +the current asset set needs the context at load time, so a worker thread can hand back a +fully GPU-resident texture. + +The main-thread coordination point still exists, but it is just `assets.Update()` draining +the channel: finished objects become *visible* to game code only on the main thread, which is +also the only place the registry is ever mutated — no locks anywhere. If a future asset type +does need context work (e.g. runtime `GenerateMips`), it can queue an action that `Update()` +executes; that seam costs nothing today. + +### Indirection for hot reload (3d) + +The registry maps `Asset` → `AssetRef`, created once and permanent (assumption 2: +loaded stays loaded, so no lifetime/refcount management at all). + +```csharp +public sealed class AssetRef +{ + public T Value { get; } // a field read, no lookup cost per use + public int Version { get; } // bumped on hot reload + // snip: internal setter used by the registry during Update() +} +``` + +Game code holds the `AssetRef` and reads `Value` when used. Derived objects (an input +layout built from a vertex shader blob) either poll `Version` or subscribe to a sparse +`Changed` event — needed rarely, mostly for shaders. + +## Hot Reload (dev only) + +``` +FileSystemWatcher (source tree) + → debounce (editors save in bursts) + → look up which assets have that file as an input (from stored input lists) + → recompile + reload on a worker, same code path as a normal load + → assets.Update() swaps AssetRef.Value on the main thread, bumps Version + → old object disposed (D3D11 keeps GPU resources alive while in flight, so this is safe) +``` + +Failures (syntax error in a shader) write the exception to the channel; `Update()` reports +it and keeps the old value live — a typo never kills the running game. + +## Project Layout + +| Project | Contents | Referenced by | +|---|---|---| +| `CapriKit.Assets` | `Asset`, registry, bundles, loaders, hot-reload swap. Refs `DirectX11`, `SuperCompressed` (transcode), `Concurrency`, `IO`. | game, always | +| `CapriKit.Assets.Compilers` | `IAssetCompiler` implementations, staleness, watcher. Refs `SuperCompressed` (encode), shader compiler. | game in dev builds; CLI tool | +| `CapriKit.Assets.Tool` | thin CLI over Compilers for the shipping build step | build scripts | + +Boilerplate budget per new asset type: **one compiler class + one loader class** +(MiniEngine3 needed processor + content wrapper + settings + serialization per type; the +shared container format and the no-settings/no-lifetime assumptions eliminate the rest). + +## Deliberately Out of Scope + +- Unloading / reference counting (assumption 2). +- Cross-asset references and cascading reloads (flat-assets decision). +- Per-asset metadata files (assumption 1). +- Packed archives — see appendix. + +## Risks & Open Points + +1. **Encoder settings vs assumption 1.** Normal maps and albedo textures ideally encode + differently (UASTC vs ETC1S, linear vs perceptual metrics). If one default proves + insufficient, a filename convention (`*_n.png`) is the escape hatch that doesn't violate + "no metadata files". +2. **Model and audio source formats** are undecided; the architecture only assumes "some + compiler produces blobs". Worth a separate research note before implementing those compilers. +3. **Input tracking in `CapriKit.IO`.** Compilers need a tracking read-only file system + (MiniEngine3's `TrackingVirtualFileSystem` is the precedent); check what `CapriKit.IO` + is missing. +4. **Native binary size.** The SuperCompressed native DLL contains encoder *and* transcoder; + shipped games carry encoder code they never call. Acceptable for now, splittable later. + +## Appendix: Packed Archives (not now, maybe later) + +Skipped because loose files make hot reload, staleness, and debugging trivially simple, and +the classic motivations (seek latency, file-handle overhead) matter little on modern SSDs. +Remaining real benefits: fewer/smaller patch artifacts, whole-archive compression, and light +obfuscation. + +The door stays open cheaply: loaders read compiled containers through +`IReadOnlyVirtualFileSystem`, so a future `PackedFileSystem` implementation (one archive = +one mounted file system) would slot in without touching any pipeline code. If it ever +happens, pack as a *post-step* over the compiled tree so the compile pipeline never knows. + +## Remark: uploading textures from a worker thread + +`UpdateSubresource`/`Map` need the immediate context (main thread), but they are only for +writing into a resource that *already exists* (`Default`/`Dynamic` usage). Asset textures +instead pass their pixels as initial data to `ID3D11Device::CreateTexture2D`, which is +free-threaded: one `SubresourceData` entry per subresource (mip × array slice), copied by +the driver *during* the create call. Use `ResourceUsage.Immutable` — it requires initial +data at creation, forbids later updates, and lets the driver optimize. + +```csharp +var subresources = new SubresourceData[mipCount]; +for (var mip = 0; mip < mipCount; mip++) +{ + // for BC formats: rowPitch = Math.Max(1, (mipWidth + 3) / 4) * bytesPerBlock + subresources[mip] = new SubresourceData(pointerToMipData, rowPitch); +} +var desc = new Texture2DDescription(format, width, height, mipLevels: mipCount, /* snip */ usage: ResourceUsage.Immutable); +var texture = device.CreateTexture2D(desc, subresources); +// pointers only need to stay pinned until CreateTexture2D returns — the copy is synchronous +``` + +Caveats: free-threaded means thread-safe, not necessarily concurrent — without +`DriverConcurrentCreates` (see `CheckFeatureSupport(D3D11_FEATURE_THREADING)`) creates +serialize on a device-wide lock, which is still correct. And none of this holds if the +device were created with `D3D11_CREATE_DEVICE_SINGLETHREADED` (ours is not, see `Device.cs`). diff --git a/source/CapriKit.AssetPipeline/AssetTranscoder.cs b/source/CapriKit.AssetPipeline/AssetTranscoder.cs new file mode 100644 index 0000000..ebf6d29 --- /dev/null +++ b/source/CapriKit.AssetPipeline/AssetTranscoder.cs @@ -0,0 +1,122 @@ +using CapriKit.IO; +using CapriKit.IO.Buffers; +using System.Buffers; +using System.IO.Pipelines; + +namespace CapriKit.AssetPipeline; + +internal sealed record Envelope(T Asset, IReadOnlySet Dependencies); + +internal static class AssetTranscoder +{ + // Encoder id (16 bytes) + encoder version (4 bytes) + payload length (4 bytes) + private const int HeaderSizeInBytes = 24; + + public static async Task Encode(AssetId id, IVirtualFileSystem fileSystem, IAssetEncoder encoder) + { + ThrowOnFileNotFound(id.Path, fileSystem); + + var outputPath = ToEncodedFilePath(id.Path); + using var output = fileSystem.CreateReadWrite(outputPath); + var writer = PipeWriter.Create(output); + + // Header + writer.Write(encoder.Id); + writer.Write(encoder.Version); + + // Payload + var payload = new ArrayBufferWriter(); + var spy = fileSystem.SpyOn(); + await encoder.Encode(id, spy, payload); + writer.Write(payload.WrittenCount); + writer.Write(payload.WrittenSpan); + + // Asset dependencies, including the source file itself + writer.Write(spy.OpenedFiles.Count); + foreach (var dependency in spy.OpenedFiles) + { + writer.Write(dependency); + } + + await writer.FlushAsync(); + await writer.CompleteAsync(); + } + + public static async Task> Decode(AssetId id, IVirtualFileSystem fileSystem, IAssetDecoder decoder) + { + var inputPath = ToEncodedFilePath(id.Path); + ThrowOnFileNotFound(inputPath, fileSystem); + + using var input = fileSystem.OpenRead(inputPath); + + // Header + var header = new byte[HeaderSizeInBytes]; + await input.ReadExactlyAsync(header); + var payloadLength = ReadHeader(header, decoder, inputPath); + + // Note: rented buffers are AT LEAST the requested length + var remainingLength = (int)(input.Length - input.Position); + var buffer = ArrayPool.Shared.Rent(remainingLength); + try + { + await input.ReadExactlyAsync(buffer.AsMemory(0, remainingLength)); + + // Payload + var payloadBuffer = buffer.AsMemory(0, payloadLength); + var payloadReader = new SequenceReader(new ReadOnlySequence(payloadBuffer)); + var asset = decoder.Decode(id, ref payloadReader); + + // Dependencies + var dependencyBuffer = buffer.AsMemory(payloadLength, remainingLength - payloadLength); + var dependencyReader = new SequenceReader(new ReadOnlySequence(dependencyBuffer)); + var dependencies = ReadDependencies(ref dependencyReader); + + return new Envelope(asset, dependencies); + } + finally + { + ArrayPool.Shared.Return(buffer); + } + } + + private static int ReadHeader(byte[] header, IAssetDecoder decoder, FilePath path) + { + var reader = new SequenceReader(new ReadOnlySequence(header)); + var id = reader.ReadGuid(); + var version = reader.ReadInt32(); + var payloadLength = reader.ReadInt32(); + + if (id != decoder.Id || version != decoder.Version) + { + throw new InvalidDataException( + $"Cannot decode {path}, it was encoded by {id} v{version} but the decoder is {decoder.Id} v{decoder.Version}"); + } + + return payloadLength; + } + + private static HashSet ReadDependencies(ref SequenceReader reader) + { + var count = reader.ReadInt32(); + var dependencies = new HashSet(count); + for (var i = 0; i < count; i++) + { + dependencies.Add(reader.ReadString()); + } + + return dependencies; + } + + private static FilePath ToEncodedFilePath(FilePath path) + { + return path + ".cka"; + } + + private static void ThrowOnFileNotFound(FilePath path, IVirtualFileSystem fileSystem) + { + if (!fileSystem.Exists(path)) + { + throw new FileNotFoundException(null, path); + } + } +} diff --git a/source/CapriKit.AssetPipeline/CapriKit.AssetPipeline.csproj b/source/CapriKit.AssetPipeline/CapriKit.AssetPipeline.csproj new file mode 100644 index 0000000..4e4824a --- /dev/null +++ b/source/CapriKit.AssetPipeline/CapriKit.AssetPipeline.csproj @@ -0,0 +1,5 @@ + + + + + diff --git a/source/CapriKit.AssetPipeline/IAssetCompiler.cs b/source/CapriKit.AssetPipeline/IAssetCompiler.cs new file mode 100644 index 0000000..6e8fb7b --- /dev/null +++ b/source/CapriKit.AssetPipeline/IAssetCompiler.cs @@ -0,0 +1,34 @@ +using CapriKit.IO; +using System.Buffers; + +namespace CapriKit.AssetPipeline; + +public record AssetId(string Key, FilePath Path); + +public interface IAssetEncoder +{ + IReadOnlySet SupportedExtensions { get; } + Guid Id { get; } + int Version { get; } + + Task Encode(AssetId id, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer); +} + +// TODO: images need settings and we could even say shaders do (the entry point) if we do not want to +// rely on AssetId.Key +public interface IAssetEncoder : IAssetEncoder +{ + Task Encode(AssetId id, TSettings settings, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer); +} + +public interface IAssetDecoder +{ + Guid Id { get; } + int Version { get; } + + // Synchronous by design: the envelope owns all file IO and hands the decoder an + // in-memory payload. The reader's buffer is only valid for the duration of the call, + // decoders must copy out anything they want to keep. + TAsset Decode(AssetId id, ref SequenceReader reader); + void HotSwap(TAsset instance, TAsset replacement); +} diff --git a/source/CapriKit.DirectX11/Assets/ITextureEncoder.cs b/source/CapriKit.DirectX11/Assets/ITextureEncoder.cs new file mode 100644 index 0000000..83694a0 --- /dev/null +++ b/source/CapriKit.DirectX11/Assets/ITextureEncoder.cs @@ -0,0 +1,10 @@ +namespace CapriKit.DirectX11.Assets; + +public interface ITextureEncoder +{ +} + +public interface ITextureDecoder +{ + +} diff --git a/source/CapriKit.DirectX11/Assets/VertexShaderTranscoder.cs b/source/CapriKit.DirectX11/Assets/VertexShaderTranscoder.cs new file mode 100644 index 0000000..6db4efb --- /dev/null +++ b/source/CapriKit.DirectX11/Assets/VertexShaderTranscoder.cs @@ -0,0 +1,44 @@ +using CapriKit.AssetPipeline; +using CapriKit.DirectX11.Resources.Shaders; +using CapriKit.IO; +using CapriKit.IO.Buffers; +using System.Buffers; + +namespace CapriKit.DirectX11.Assets; + +public sealed class VertexShaderTranscoder(Device device) : IAssetEncoder, IAssetDecoder +{ + public IReadOnlySet SupportedExtensions { get; } = new HashSet([".hlsl"]); + public Guid Id => Guid.Parse("{CA3CB37D-9880-4B61-AB09-EBC17E7533E6}"); + public int Version => 1; + + public async Task Encode(AssetId id, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer) + { + var source = await fileSystem.ReadAllText(id.Path); + var includePath = id.Path.Directory; + var bytes = ShaderCompiler.CompileVertexShader(fileSystem, includePath, source, id.Key, id.ToString()); + + writer.Write(bytes.EntryPoint); + writer.Write(bytes.Name); + writer.Write(bytes.Bytes.Length); + writer.Write(bytes.Bytes); + } + + public IVertexShader Decode(AssetId id, ref SequenceReader reader) + { + var entryPoint = reader.ReadString(); + var name = reader.ReadString(); + var length = reader.ReadInt32(); + var bytes = reader.ReadBytes(length); + + var byteCode = new VertexShaderByteCode(bytes, entryPoint, name); + return ShaderCompiler.CreateVertexShader(byteCode, device); + } + + public void HotSwap(IVertexShader instance, IVertexShader replacement) + { + var original = instance.ID3D11VertexShader; + instance.ID3D11VertexShader = replacement.ID3D11VertexShader; + original.Dispose(); + } +} diff --git a/source/CapriKit.DirectX11/CapriKit.DirectX11.csproj b/source/CapriKit.DirectX11/CapriKit.DirectX11.csproj index d4db051..8e565dc 100644 --- a/source/CapriKit.DirectX11/CapriKit.DirectX11.csproj +++ b/source/CapriKit.DirectX11/CapriKit.DirectX11.csproj @@ -8,6 +8,7 @@ + diff --git a/source/CapriKit.DirectX11/Resources/Shaders/VertexShader.cs b/source/CapriKit.DirectX11/Resources/Shaders/VertexShader.cs index 39ccfb7..a26b072 100644 --- a/source/CapriKit.DirectX11/Resources/Shaders/VertexShader.cs +++ b/source/CapriKit.DirectX11/Resources/Shaders/VertexShader.cs @@ -4,15 +4,15 @@ namespace CapriKit.DirectX11.Resources.Shaders; public interface IVertexShader : IDisposable { - internal ID3D11VertexShader ID3D11VertexShader { get; } + internal ID3D11VertexShader ID3D11VertexShader { get; set; } - IInputLayout CreateInputLayout(Device device, InputElementDescription[] elements); // TODO: How to do this without exposing types from Vortice? + IInputLayout CreateInputLayout(Device device, InputElementDescription[] elements); } internal sealed class VertexShader : IVertexShader { private readonly byte[] Blob; - private readonly ID3D11VertexShader Shader; + private ID3D11VertexShader Shader; internal VertexShader(byte[] blob, ID3D11VertexShader shader) { @@ -20,7 +20,11 @@ internal VertexShader(byte[] blob, ID3D11VertexShader shader) Shader = shader; } - ID3D11VertexShader IVertexShader.ID3D11VertexShader => Shader; + ID3D11VertexShader IVertexShader.ID3D11VertexShader + { + get { return Shader; } + set { Shader = value; } + } public IInputLayout CreateInputLayout(Device device, InputElementDescription[] elements) { diff --git a/source/CapriKit.IO/Buffers/IBufferWriterExtensions.cs b/source/CapriKit.IO/Buffers/IBufferWriterExtensions.cs new file mode 100644 index 0000000..f84b3f6 --- /dev/null +++ b/source/CapriKit.IO/Buffers/IBufferWriterExtensions.cs @@ -0,0 +1,60 @@ +using System.Buffers; +using System.Buffers.Binary; +using System.Runtime.CompilerServices; +using System.Text; + +namespace CapriKit.IO.Buffers; + +public static class BufferWriterExtensions +{ + /// + /// Writes a length prefixed string. The prefix is the byte count of the encoded string as a + /// 7 bit encoded integer, identical to + /// + /// The writer to write the string to + /// The string to write + /// Defaults to UTF8 + public static void Write(this IBufferWriter writer, string value, Encoding? encoding = null) + { + encoding = encoding ?? Encoding.UTF8; + var length = encoding.GetByteCount(value); + writer.Write7BitEncodedInt(length); + encoding.GetBytes(value, writer); + } + + /// + /// Writes an integer seven bits at a time, using the eighth bit to indicate that another byte + /// follows. The format is identical to + /// + public static void Write7BitEncodedInt(this IBufferWriter writer, int value) + { + const int MaxLengthInBytes = 5; // ceil(32 bits / 7 bits per byte) + + var span = writer.GetSpan(MaxLengthInBytes); + var written = 0; + + var uValue = (uint)value; + while (uValue > 0x7Fu) + { + span[written++] = (byte)(uValue | ~0x7Fu); + uValue >>= 7; + } + span[written++] = (byte)uValue; + + writer.Advance(written); + } + + public static void Write(this IBufferWriter writer, int value) + { + var span = writer.GetSpan(sizeof(int)); + BinaryPrimitives.WriteInt32LittleEndian(span, value); + writer.Advance(sizeof(int)); + } + + public static void Write(this IBufferWriter writer, Guid guid) + { + var span = writer.GetSpan(Unsafe.SizeOf()); + guid.TryWriteBytes(span, false, out var written); + writer.Advance(written); + } +} diff --git a/source/CapriKit.IO/Buffers/SequenceReaderExtensions.cs b/source/CapriKit.IO/Buffers/SequenceReaderExtensions.cs new file mode 100644 index 0000000..da6baf6 --- /dev/null +++ b/source/CapriKit.IO/Buffers/SequenceReaderExtensions.cs @@ -0,0 +1,101 @@ +using System.Buffers; +using System.Runtime.CompilerServices; +using System.Text; + +namespace CapriKit.IO.Buffers; + +public static class SequenceReaderExtensions +{ + /// + /// Reads a length prefixed string written by + /// . + /// The format is identical to so both can be mixed + /// freely, as long as both sides use the same encoding + /// + /// The reader to read the string from + /// Defaults to UTF8 + public static string ReadString(this ref SequenceReader reader, Encoding? encoding = null) + { + encoding = encoding ?? Encoding.UTF8; + var length = reader.Read7BitEncodedInt(); + if (!reader.TryReadExact(length, out var sequence)) + { + throw new EndOfStreamException(); + } + + return encoding.GetString(in sequence); + } + + /// + /// Reads an integer written seven bits at a time by + /// + /// or + /// + public static int Read7BitEncodedInt(this ref SequenceReader reader) + { + const int MaxBytesWithoutOverflow = 4; + + var result = 0u; + for (var shift = 0; shift < MaxBytesWithoutOverflow * 7; shift += 7) + { + var current = ReadByte(ref reader); + result |= (current & 0x7Fu) << shift; + if (current <= 0x7Fu) + { + return (int)result; + } + } + + // The fifth byte can only hold the 4 remaining bits of a 32 bit integer + var last = ReadByte(ref reader); + if (last > 0b_1111u) + { + throw new FormatException("Invalid 7 bit encoded integer"); + } + + result |= (uint)last << (MaxBytesWithoutOverflow * 7); + return (int)result; + } + + public static int ReadInt32(this ref SequenceReader reader) + { + if (!reader.TryReadLittleEndian(out int value)) + { + throw new EndOfStreamException(); + } + + return value; + } + + public static Guid ReadGuid(this ref SequenceReader reader) + { + Span bytes = stackalloc byte[Unsafe.SizeOf()]; + if (!reader.TryCopyTo(bytes)) + { + throw new EndOfStreamException(); + } + + reader.Advance(bytes.Length); + return new Guid(bytes, bigEndian: false); + } + + public static byte[] ReadBytes(this ref SequenceReader reader, int length) + { + if (!reader.TryReadExact(length, out var sequence)) + { + throw new EndOfStreamException(); + } + + return sequence.ToArray(); + } + + public static byte ReadByte(ref SequenceReader reader) + { + if (!reader.TryRead(out var value)) + { + throw new EndOfStreamException(); + } + + return value; + } +} diff --git a/source/CapriKit.IO/CapriKit.IO.csproj b/source/CapriKit.IO/CapriKit.IO.csproj index c369e11..2ae79d6 100644 --- a/source/CapriKit.IO/CapriKit.IO.csproj +++ b/source/CapriKit.IO/CapriKit.IO.csproj @@ -1,3 +1 @@ - - - + diff --git a/source/CapriKit.Tests/AssetPipeline/AssetTranscoderTests.cs b/source/CapriKit.Tests/AssetPipeline/AssetTranscoderTests.cs new file mode 100644 index 0000000..bc152cc --- /dev/null +++ b/source/CapriKit.Tests/AssetPipeline/AssetTranscoderTests.cs @@ -0,0 +1,82 @@ +using CapriKit.AssetPipeline; +using CapriKit.IO; +using CapriKit.IO.Buffers; +using System.Buffers; + +namespace CapriKit.Tests.AssetPipeline; + +internal class AssetTranscoderTests +{ + + [Test] + public async Task Decode() + { + var fileSystem = new InMemoryFileSystem(); + await fileSystem.WriteAllText("hello.txt", "héllo"); + var transcoder = new DummyTranscoder(); + var id = new AssetId("Main", "hello.txt"); + + await AssetTranscoder.Encode(id, fileSystem, transcoder); + var envelope = await AssetTranscoder.Decode(id, fileSystem, transcoder); + + FilePath expectedDependency = "hello.txt"; + await Assert.That(envelope.Asset).IsEqualTo("HÉLLO"); + await Assert.That(envelope.Dependencies.Count).IsEqualTo(1); + await Assert.That(envelope.Dependencies.First()).IsEqualTo(expectedDependency); + } + + [Test] + public async Task Encode() + { + var fileSystem = new InMemoryFileSystem(); + await fileSystem.WriteAllText("hello.txt", "héllo"); + var transcoder = new DummyTranscoder(); + var id = new AssetId("Main", "hello.txt"); + + await AssetTranscoder.Encode(id, fileSystem, transcoder); + var bytes = await fileSystem.ReadAllBytes("hello.txt.cka"); + + // SequenceReader is a ref struct, so all reading happens before the first + // await and only plain locals cross into the assertions + var reader = new SequenceReader(new ReadOnlySequence(bytes)); + var encoderId = reader.ReadGuid(); + var encoderVersion = reader.ReadInt32(); + var payloadLength = reader.ReadInt32(); + reader.Advance(payloadLength); + var dependencyCount = reader.ReadInt32(); + var dependency = reader.ReadString(); + var end = reader.End; + + await Assert.That(encoderId).IsEqualTo(transcoder.Id); + await Assert.That(encoderVersion).IsEqualTo(transcoder.Version); + await Assert.That(dependencyCount).IsEqualTo(1); + await Assert.That(dependency).IsEqualTo("hello.txt"); + await Assert.That(end).IsTrue(); + } + + /// + /// Uppercases a text file + /// + private sealed class DummyTranscoder : IAssetEncoder, IAssetDecoder + { + public IReadOnlySet SupportedExtensions { get; } = new HashSet([".txt"]); + public Guid Id => Guid.Parse("{B87F41E3-6C33-46E4-802A-3E1E82800E7A}"); + public int Version => 1; + + public async Task Encode(AssetId id, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer) + { + var text = await fileSystem.ReadAllText(id.Path); + writer.Write(text.ToUpperInvariant()); + } + + public string Decode(AssetId id, ref SequenceReader reader) + { + return reader.ReadString(); + } + + public void HotSwap(string instance, string replacement) + { + throw new NotImplementedException(); + } + } +} diff --git a/source/CapriKit.Tests/CapriKit.Tests.csproj b/source/CapriKit.Tests/CapriKit.Tests.csproj index c907671..99a7409 100644 --- a/source/CapriKit.Tests/CapriKit.Tests.csproj +++ b/source/CapriKit.Tests/CapriKit.Tests.csproj @@ -15,6 +15,7 @@ + diff --git a/source/CapriKit.Tests/IO/Buffers/IBufferWriterExtensionsTests.cs b/source/CapriKit.Tests/IO/Buffers/IBufferWriterExtensionsTests.cs new file mode 100644 index 0000000..f0bd49d --- /dev/null +++ b/source/CapriKit.Tests/IO/Buffers/IBufferWriterExtensionsTests.cs @@ -0,0 +1,76 @@ +using CapriKit.IO.Buffers; +using System.Buffers; +using System.Buffers.Binary; +using System.Text; + +namespace CapriKit.Tests.IO.Buffers; + +internal class IBufferWriterExtensionsTests +{ + [Test] + public async Task Write_Int32() + { + var writer = new ArrayBufferWriter(); + + writer.Write(-12345); + + var bytes = writer.WrittenMemory.ToArray(); + var value = BinaryPrimitives.ReadInt32LittleEndian(bytes); + + await Assert.That(bytes.Length).IsEqualTo(sizeof(int)); + await Assert.That(value).IsEqualTo(-12345); + } + + [Test] + public async Task Write_String() + { + var writer = new ArrayBufferWriter(); + var value = "héllo"; // 'é' encodes to two bytes, so the prefix must count bytes, not chars + + writer.Write(value); + + // The format promises to be identical to BinaryWriter/BinaryReader's + using var stream = new MemoryStream(writer.WrittenMemory.ToArray()); + using var reader = new BinaryReader(stream, Encoding.UTF8); + var text = reader.ReadString(); + + await Assert.That(text).IsEqualTo(value); + await Assert.That(stream.Position).IsEqualTo(stream.Length); + } + + [Test] + public async Task Write7BitEncodedInt() + { + int[] values = [0, 127, 128, 300, int.MaxValue, -1]; + var writer = new ArrayBufferWriter(); + foreach (var value in values) + { + writer.Write7BitEncodedInt(value); + } + + // The format promises to be identical to BinaryWriter/BinaryReader's + using var stream = new MemoryStream(writer.WrittenMemory.ToArray()); + using var reader = new BinaryReader(stream, Encoding.UTF8); + foreach (var value in values) + { + await Assert.That(reader.Read7BitEncodedInt()).IsEqualTo(value); + } + + await Assert.That(stream.Position).IsEqualTo(stream.Length); + } + + [Test] + public async Task Write_Guid() + { + var writer = new ArrayBufferWriter(); + var guid = Guid.NewGuid(); + + writer.Write(guid); + + var bytes = writer.WrittenMemory.ToArray(); + var roundTripped = new Guid(bytes, bigEndian: false); + + await Assert.That(bytes.Length).IsEqualTo(16); + await Assert.That(roundTripped).IsEqualTo(guid); + } +} diff --git a/source/CapriKit.Tests/IO/Buffers/SequenceReaderExtensionsTests.cs b/source/CapriKit.Tests/IO/Buffers/SequenceReaderExtensionsTests.cs new file mode 100644 index 0000000..bb7ef01 --- /dev/null +++ b/source/CapriKit.Tests/IO/Buffers/SequenceReaderExtensionsTests.cs @@ -0,0 +1,109 @@ +using CapriKit.IO.Buffers; +using System.Buffers; +using System.Text; + +namespace CapriKit.Tests.IO.Buffers; + +internal class SequenceReaderExtensionsTests +{ + // SequenceReader is a ref struct, so all reading happens before the first + // await and only plain locals cross into the assertions + + [Test] + public async Task ReadInt32() + { + var writer = new ArrayBufferWriter(); + writer.Write(-12345); + + var reader = new SequenceReader(new ReadOnlySequence(writer.WrittenMemory)); + var value = reader.ReadInt32(); + var end = reader.End; + + await Assert.That(value).IsEqualTo(-12345); + await Assert.That(end).IsTrue(); + } + + [Test] + public async Task ReadString() + { + var writer = new ArrayBufferWriter(); + writer.Write("héllo"); // 'é' encodes to two bytes, exercising the bytes-not-chars length prefix + + var reader = new SequenceReader(new ReadOnlySequence(writer.WrittenMemory)); + var value = reader.ReadString(); + var end = reader.End; + + await Assert.That(value).IsEqualTo("héllo"); + await Assert.That(end).IsTrue(); + } + + [Test] + public async Task ReadString_WrittenByBinaryWriter() + { + using var stream = new MemoryStream(); + using (var writer = new BinaryWriter(stream, Encoding.UTF8, leaveOpen: true)) + { + writer.Write("héllo"); + } + + var reader = new SequenceReader(new ReadOnlySequence(stream.ToArray())); + var value = reader.ReadString(); + var end = reader.End; + + await Assert.That(value).IsEqualTo("héllo"); + await Assert.That(end).IsTrue(); + } + + [Test] + public async Task Read7BitEncodedInt() + { + int[] values = [0, 127, 128, 300, int.MaxValue, -1]; + using var stream = new MemoryStream(); + using (var writer = new BinaryWriter(stream, Encoding.UTF8, leaveOpen: true)) + { + foreach (var value in values) + { + writer.Write7BitEncodedInt(value); + } + } + + var reader = new SequenceReader(new ReadOnlySequence(stream.ToArray())); + var results = new List(); + while (!reader.End) + { + results.Add(reader.Read7BitEncodedInt()); + } + + await Assert.That(results.SequenceEqual(values)).IsTrue(); + } + + [Test] + public async Task ReadGuid() + { + var guid = Guid.NewGuid(); + var writer = new ArrayBufferWriter(); + writer.Write(guid); + + var reader = new SequenceReader(new ReadOnlySequence(writer.WrittenMemory)); + var value = reader.ReadGuid(); + var end = reader.End; + + await Assert.That(value).IsEqualTo(guid); + await Assert.That(end).IsTrue(); + } + + [Test] + public async Task ReadBytes() + { + var bytes = new byte[] { 1, 2, 3, 4, 5 }; + var writer = new ArrayBufferWriter(); + writer.Write(bytes); + + var reader = new SequenceReader(new ReadOnlySequence(writer.WrittenMemory)); + var value = reader.ReadBytes(bytes.Length); + var end = reader.End; + + await Assert.That(value.SequenceEqual(bytes)).IsTrue(); + await Assert.That(end).IsTrue(); + } +} From dcc0d3a6db5af44778c23b79d2f88936f69376d1 Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Sat, 11 Jul 2026 17:17:51 +0200 Subject: [PATCH 03/53] Nicer decoding --- .../CapriKit.AssetPipeline/AssetTranscoder.cs | 89 +++++++++++-------- .../Buffers/SequenceReaderExtensions.cs | 11 +++ source/CapriKit.IO/FileSystem.cs | 39 +++++--- source/CapriKit.IO/FileSystemEventListener.cs | 6 +- .../IO/FileSystemEventListenerTests.cs | 3 +- 5 files changed, 91 insertions(+), 57 deletions(-) diff --git a/source/CapriKit.AssetPipeline/AssetTranscoder.cs b/source/CapriKit.AssetPipeline/AssetTranscoder.cs index ebf6d29..eb5b63d 100644 --- a/source/CapriKit.AssetPipeline/AssetTranscoder.cs +++ b/source/CapriKit.AssetPipeline/AssetTranscoder.cs @@ -7,6 +7,7 @@ namespace CapriKit.AssetPipeline; internal sealed record Envelope(T Asset, IReadOnlySet Dependencies); +// TODO: split in encoder and decoder and make the encoder as nice as the decoder internal static class AssetTranscoder { // Encoder id (16 bytes) + encoder version (4 bytes) + payload length (4 bytes) @@ -49,29 +50,31 @@ public static async Task> Decode(AssetId id, IVirtualFileSystem f using var input = fileSystem.OpenRead(inputPath); - // Header - var header = new byte[HeaderSizeInBytes]; - await input.ReadExactlyAsync(header); - var payloadLength = ReadHeader(header, decoder, inputPath); + var payloadLength = await ReadHeader(input, decoder, inputPath); + var asset = await ReadPayload(input, payloadLength, decoder, id); + var dependencies = await ReadDependencies(input); + + return new Envelope(asset, dependencies); + } - // Note: rented buffers are AT LEAST the requested length - var remainingLength = (int)(input.Length - input.Position); - var buffer = ArrayPool.Shared.Rent(remainingLength); + private static async Task ReadHeader(Stream input, IAssetDecoder decoder, FilePath path) + { + var buffer = ArrayPool.Shared.Rent(HeaderSizeInBytes); try { - await input.ReadExactlyAsync(buffer.AsMemory(0, remainingLength)); - - // Payload - var payloadBuffer = buffer.AsMemory(0, payloadLength); - var payloadReader = new SequenceReader(new ReadOnlySequence(payloadBuffer)); - var asset = decoder.Decode(id, ref payloadReader); - - // Dependencies - var dependencyBuffer = buffer.AsMemory(payloadLength, remainingLength - payloadLength); - var dependencyReader = new SequenceReader(new ReadOnlySequence(dependencyBuffer)); - var dependencies = ReadDependencies(ref dependencyReader); - - return new Envelope(asset, dependencies); + await input.ReadExactlyAsync(buffer.AsMemory(0, HeaderSizeInBytes)); + var reader = SequenceReaders.Create(buffer, 0, HeaderSizeInBytes); + var id = reader.ReadGuid(); + var version = reader.ReadInt32(); + var payloadLength = reader.ReadInt32(); + + if (id != decoder.Id || version != decoder.Version) + { + throw new InvalidDataException( + $"Cannot decode {path}, it was encoded by {id} v{version} but the decoder is {decoder.Id} v{decoder.Version}"); + } + + return payloadLength; } finally { @@ -79,32 +82,42 @@ public static async Task> Decode(AssetId id, IVirtualFileSystem f } } - private static int ReadHeader(byte[] header, IAssetDecoder decoder, FilePath path) + private static async Task ReadPayload(Stream input, int payloadLength, IAssetDecoder decoder, AssetId id) { - var reader = new SequenceReader(new ReadOnlySequence(header)); - var id = reader.ReadGuid(); - var version = reader.ReadInt32(); - var payloadLength = reader.ReadInt32(); - - if (id != decoder.Id || version != decoder.Version) + var buffer = ArrayPool.Shared.Rent(payloadLength); + try { - throw new InvalidDataException( - $"Cannot decode {path}, it was encoded by {id} v{version} but the decoder is {decoder.Id} v{decoder.Version}"); + await input.ReadExactlyAsync(buffer.AsMemory(0, payloadLength)); + var reader = SequenceReaders.Create(buffer, 0, payloadLength); + return decoder.Decode(id, ref reader); + } + finally + { + ArrayPool.Shared.Return(buffer); } - - return payloadLength; } - private static HashSet ReadDependencies(ref SequenceReader reader) + private static async Task> ReadDependencies(Stream input) { - var count = reader.ReadInt32(); - var dependencies = new HashSet(count); - for (var i = 0; i < count; i++) + var length = (int)(input.Length - input.Position); + var buffer = ArrayPool.Shared.Rent(length); + try { - dependencies.Add(reader.ReadString()); + await input.ReadExactlyAsync(buffer.AsMemory(0, length)); + var reader = SequenceReaders.Create(buffer, 0, length); + + var count = reader.ReadInt32(); + var dependencies = new HashSet(count); + for (var i = 0; i < count; i++) + { + dependencies.Add(reader.ReadString()); + } + return dependencies; + } + finally + { + ArrayPool.Shared.Return(buffer); } - - return dependencies; } private static FilePath ToEncodedFilePath(FilePath path) diff --git a/source/CapriKit.IO/Buffers/SequenceReaderExtensions.cs b/source/CapriKit.IO/Buffers/SequenceReaderExtensions.cs index da6baf6..3ea73b9 100644 --- a/source/CapriKit.IO/Buffers/SequenceReaderExtensions.cs +++ b/source/CapriKit.IO/Buffers/SequenceReaderExtensions.cs @@ -4,8 +4,19 @@ namespace CapriKit.IO.Buffers; +public static class SequenceReaders +{ + public static SequenceReader Create(byte[] bytes, int start, int length) + { + var sequence = new ReadOnlySequence(bytes, start, length); + return new SequenceReader(sequence); + } +} + public static class SequenceReaderExtensions { + + /// /// Reads a length prefixed string written by /// . diff --git a/source/CapriKit.IO/FileSystem.cs b/source/CapriKit.IO/FileSystem.cs index 49ad138..750e919 100644 --- a/source/CapriKit.IO/FileSystem.cs +++ b/source/CapriKit.IO/FileSystem.cs @@ -5,7 +5,14 @@ public class FileSystem : IVirtualFileSystem public Stream AppendWrite(FilePath file) { var absoluteFile = FindOrThrow(file); - return absoluteFile.Open(FileMode.Append, FileAccess.Write, FileShare.Read); + var options = new FileStreamOptions() + { + Mode = FileMode.Append, + Access = FileAccess.Write, + Share = FileShare.Read, + Options = FileOptions.Asynchronous + }; + return new FileStream(absoluteFile.FullName, options); } public Stream CreateReadWrite(FilePath file) @@ -16,8 +23,14 @@ public Stream CreateReadWrite(FilePath file) var directory = absoluteFile.DirectoryName ?? string.Empty; Directory.CreateDirectory(directory); } - - return absoluteFile.Open(FileMode.Create, FileAccess.ReadWrite, FileShare.Read); + var options = new FileStreamOptions() + { + Mode = FileMode.Create, + Access = FileAccess.ReadWrite, + Share = FileShare.Read, + Options = FileOptions.Asynchronous + }; + return new FileStream(absoluteFile.FullName, options); } public void Delete(FilePath file) @@ -41,7 +54,14 @@ public DateTime LastWriteTime(FilePath file) public Stream OpenRead(FilePath file) { var absoluteFile = FindOrThrow(file); - return absoluteFile.Open(FileMode.Open, FileAccess.Read, FileShare.Read); + var options = new FileStreamOptions() + { + Mode = FileMode.Open, + Access = FileAccess.Read, + Share = FileShare.Read, + Options = FileOptions.Asynchronous + }; + return new FileStream(absoluteFile.FullName, options); } public long SizeInBytes(FilePath file) @@ -65,11 +85,6 @@ public IReadOnlyList List(DirectoryPath directory) return filePaths; } - public FileSystemEventListener Watch(DirectoryPath directory, bool includeSubDirectories = true) - { - return new FileSystemEventListener(this, directory, includeSubDirectories); - } - private FileInfo FindOrThrow(FilePath file) { var info = GetFileInfo(file); @@ -81,18 +96,18 @@ private FileInfo FindOrThrow(FilePath file) throw new FileNotFoundException(null, file.ToString()); } - internal FilePath GetFilePath(string path) + internal static FilePath GetFilePath(string path) { return new FilePath(path); } - internal FileInfo GetFileInfo(FilePath file) + internal static FileInfo GetFileInfo(FilePath file) { var absolutePath = file.IsAbsolute ? file : file.ToAbsolute(); return new FileInfo(absolutePath.ToString()); } - internal DirectoryInfo GetDirectoryInfo(DirectoryPath path) + internal static DirectoryInfo GetDirectoryInfo(DirectoryPath path) { var absolutePath = path.IsAbsolute ? path : path.ToAbsolute(); return new DirectoryInfo(absolutePath.ToString()); diff --git a/source/CapriKit.IO/FileSystemEventListener.cs b/source/CapriKit.IO/FileSystemEventListener.cs index edc9f0f..241928a 100644 --- a/source/CapriKit.IO/FileSystemEventListener.cs +++ b/source/CapriKit.IO/FileSystemEventListener.cs @@ -11,17 +11,13 @@ public enum FileSystemChangeKind public sealed class FileSystemEventListener : IDisposable { - private readonly FileSystem FileSystem; private readonly FileSystemWatcher Watcher; private event FileSystemEventHandler? onFileChanged; - public FileSystemEventListener(FileSystem fileSystem, DirectoryPath directory, bool includeSubDirectories = true) + public FileSystemEventListener(DirectoryPath directory, bool includeSubDirectories = true) { - FileSystem = fileSystem; Directory = directory; - - // Take the directory info since all file system types will point that to the true absolute path var directoryInfo = FileSystem.GetDirectoryInfo(directory); if (!directoryInfo.Exists) { diff --git a/source/CapriKit.Tests/IO/FileSystemEventListenerTests.cs b/source/CapriKit.Tests/IO/FileSystemEventListenerTests.cs index d991854..cbfd980 100644 --- a/source/CapriKit.Tests/IO/FileSystemEventListenerTests.cs +++ b/source/CapriKit.Tests/IO/FileSystemEventListenerTests.cs @@ -30,10 +30,9 @@ public async Task OnFileChanged() var changed = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); var deleted = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); - var fileSystem = new FileSystem(); var scopedFileSystem = new ScopedFileSystem(fileSystem, TempDirectory); - using var watcher = fileSystem.Watch(TempDirectory); + using var watcher = new FileSystemEventListener(TempDirectory, false); watcher.OnFileChanged += (s, e) => { if (e.reason == FileSystemChangeKind.Created && e.target.FileName.Equals(fileName, StringComparison.OrdinalIgnoreCase)) From de97a1e0390f70d4d2ba148233eb3ba350f905b6 Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Mon, 13 Jul 2026 19:58:17 +0200 Subject: [PATCH 04/53] Nicer encoding --- .../{AssetTranscoder.cs => AssetDecoder.cs} | 56 ++----------- source/CapriKit.AssetPipeline/AssetEncoder.cs | 52 ++++++++++++ .../CapriKit.AssetPipeline/AssetUtilities.cs | 19 +++++ source/CapriKit.AssetPipeline/Envelope.cs | 5 ++ .../AssetPipeline/AssetDecoderTests.cs | 25 ++++++ .../AssetPipeline/AssetEncoderTests.cs | 36 ++++++++ .../AssetPipeline/AssetTranscoderTests.cs | 82 ------------------- .../AssetPipeline/DummyTranscoder.cs | 32 ++++++++ 8 files changed, 174 insertions(+), 133 deletions(-) rename source/CapriKit.AssetPipeline/{AssetTranscoder.cs => AssetDecoder.cs} (61%) create mode 100644 source/CapriKit.AssetPipeline/AssetEncoder.cs create mode 100644 source/CapriKit.AssetPipeline/AssetUtilities.cs create mode 100644 source/CapriKit.AssetPipeline/Envelope.cs create mode 100644 source/CapriKit.Tests/AssetPipeline/AssetDecoderTests.cs create mode 100644 source/CapriKit.Tests/AssetPipeline/AssetEncoderTests.cs delete mode 100644 source/CapriKit.Tests/AssetPipeline/AssetTranscoderTests.cs create mode 100644 source/CapriKit.Tests/AssetPipeline/DummyTranscoder.cs diff --git a/source/CapriKit.AssetPipeline/AssetTranscoder.cs b/source/CapriKit.AssetPipeline/AssetDecoder.cs similarity index 61% rename from source/CapriKit.AssetPipeline/AssetTranscoder.cs rename to source/CapriKit.AssetPipeline/AssetDecoder.cs index eb5b63d..40eda84 100644 --- a/source/CapriKit.AssetPipeline/AssetTranscoder.cs +++ b/source/CapriKit.AssetPipeline/AssetDecoder.cs @@ -1,48 +1,13 @@ using CapriKit.IO; using CapriKit.IO.Buffers; using System.Buffers; -using System.IO.Pipelines; -namespace CapriKit.AssetPipeline; +using static CapriKit.AssetPipeline.AssetUtilities; -internal sealed record Envelope(T Asset, IReadOnlySet Dependencies); +namespace CapriKit.AssetPipeline; -// TODO: split in encoder and decoder and make the encoder as nice as the decoder -internal static class AssetTranscoder +internal static class AssetDecoder { - // Encoder id (16 bytes) + encoder version (4 bytes) + payload length (4 bytes) - private const int HeaderSizeInBytes = 24; - - public static async Task Encode(AssetId id, IVirtualFileSystem fileSystem, IAssetEncoder encoder) - { - ThrowOnFileNotFound(id.Path, fileSystem); - - var outputPath = ToEncodedFilePath(id.Path); - using var output = fileSystem.CreateReadWrite(outputPath); - var writer = PipeWriter.Create(output); - - // Header - writer.Write(encoder.Id); - writer.Write(encoder.Version); - - // Payload - var payload = new ArrayBufferWriter(); - var spy = fileSystem.SpyOn(); - await encoder.Encode(id, spy, payload); - writer.Write(payload.WrittenCount); - writer.Write(payload.WrittenSpan); - - // Asset dependencies, including the source file itself - writer.Write(spy.OpenedFiles.Count); - foreach (var dependency in spy.OpenedFiles) - { - writer.Write(dependency); - } - - await writer.FlushAsync(); - await writer.CompleteAsync(); - } - public static async Task> Decode(AssetId id, IVirtualFileSystem fileSystem, IAssetDecoder decoder) { var inputPath = ToEncodedFilePath(id.Path); @@ -59,6 +24,8 @@ public static async Task> Decode(AssetId id, IVirtualFileSystem f private static async Task ReadHeader(Stream input, IAssetDecoder decoder, FilePath path) { + // Encoder id (16 bytes) + encoder version (4 bytes) + payload length (4 bytes) + const int HeaderSizeInBytes = 24; var buffer = ArrayPool.Shared.Rent(HeaderSizeInBytes); try { @@ -119,17 +86,4 @@ private static async Task> ReadDependencies(Stream input) ArrayPool.Shared.Return(buffer); } } - - private static FilePath ToEncodedFilePath(FilePath path) - { - return path + ".cka"; - } - - private static void ThrowOnFileNotFound(FilePath path, IVirtualFileSystem fileSystem) - { - if (!fileSystem.Exists(path)) - { - throw new FileNotFoundException(null, path); - } - } } diff --git a/source/CapriKit.AssetPipeline/AssetEncoder.cs b/source/CapriKit.AssetPipeline/AssetEncoder.cs new file mode 100644 index 0000000..59fb412 --- /dev/null +++ b/source/CapriKit.AssetPipeline/AssetEncoder.cs @@ -0,0 +1,52 @@ +using CapriKit.IO; +using CapriKit.IO.Buffers; +using System.Buffers; +using System.IO.Pipelines; +using static CapriKit.AssetPipeline.AssetUtilities; + +namespace CapriKit.AssetPipeline; + +internal sealed class AssetEncoder +{ + public static async Task Encode(AssetId id, IVirtualFileSystem fileSystem, IAssetEncoder encoder) + { + ThrowOnFileNotFound(id.Path, fileSystem); + var outputPath = ToEncodedFilePath(id.Path); + + using var output = fileSystem.CreateReadWrite(outputPath); + var writer = PipeWriter.Create(output); + var spy = fileSystem.SpyOn(); + + WriteHeader(writer, encoder); // payload length is written in WritePayload + await WritePayload(id, encoder, writer, spy); + WriteDependencies(spy, writer); + + await writer.FlushAsync(); + await writer.CompleteAsync(); + } + + private static void WriteHeader(PipeWriter writer, IAssetEncoder encoder) + { + writer.Write(encoder.Id); + writer.Write(encoder.Version); + } + + private static async Task WritePayload(AssetId id, IAssetEncoder encoder, PipeWriter writer, VirtualFileSystemSpy spy) + { + var payload = new ArrayBufferWriter(); + await encoder.Encode(id, spy, payload); + writer.Write(payload.WrittenCount); + writer.Write(payload.WrittenSpan); + } + + private static void WriteDependencies(VirtualFileSystemSpy spy, PipeWriter writer) + { + writer.Write(spy.OpenedFiles.Count); + foreach (var dependency in spy.OpenedFiles) + { + writer.Write(dependency); + } + } + + +} diff --git a/source/CapriKit.AssetPipeline/AssetUtilities.cs b/source/CapriKit.AssetPipeline/AssetUtilities.cs new file mode 100644 index 0000000..5999b10 --- /dev/null +++ b/source/CapriKit.AssetPipeline/AssetUtilities.cs @@ -0,0 +1,19 @@ +using CapriKit.IO; + +namespace CapriKit.AssetPipeline; + +internal static class AssetUtilities +{ + public static FilePath ToEncodedFilePath(FilePath path) + { + return path + ".cka"; + } + + public static void ThrowOnFileNotFound(FilePath path, IVirtualFileSystem fileSystem) + { + if (!fileSystem.Exists(path)) + { + throw new FileNotFoundException(null, path); + } + } +} diff --git a/source/CapriKit.AssetPipeline/Envelope.cs b/source/CapriKit.AssetPipeline/Envelope.cs new file mode 100644 index 0000000..3cf8129 --- /dev/null +++ b/source/CapriKit.AssetPipeline/Envelope.cs @@ -0,0 +1,5 @@ +using CapriKit.IO; + +namespace CapriKit.AssetPipeline; + +internal sealed record Envelope(T Asset, IReadOnlySet Dependencies); diff --git a/source/CapriKit.Tests/AssetPipeline/AssetDecoderTests.cs b/source/CapriKit.Tests/AssetPipeline/AssetDecoderTests.cs new file mode 100644 index 0000000..786d2f7 --- /dev/null +++ b/source/CapriKit.Tests/AssetPipeline/AssetDecoderTests.cs @@ -0,0 +1,25 @@ +using CapriKit.AssetPipeline; +using CapriKit.IO; + +namespace CapriKit.Tests.AssetPipeline; + +internal class AssetDecoderTests +{ + + [Test] + public async Task Decode() + { + var fileSystem = new InMemoryFileSystem(); + await fileSystem.WriteAllText("hello.txt", "héllo"); + var transcoder = new DummyTranscoder(); + var id = new AssetId("Main", "hello.txt"); + + await AssetEncoder.Encode(id, fileSystem, transcoder); + var envelope = await AssetDecoder.Decode(id, fileSystem, transcoder); + + FilePath expectedDependency = "hello.txt"; + await Assert.That(envelope.Asset).IsEqualTo("HÉLLO"); + await Assert.That(envelope.Dependencies.Count).IsEqualTo(1); + await Assert.That(envelope.Dependencies.First()).IsEqualTo(expectedDependency); + } +} diff --git a/source/CapriKit.Tests/AssetPipeline/AssetEncoderTests.cs b/source/CapriKit.Tests/AssetPipeline/AssetEncoderTests.cs new file mode 100644 index 0000000..df49367 --- /dev/null +++ b/source/CapriKit.Tests/AssetPipeline/AssetEncoderTests.cs @@ -0,0 +1,36 @@ +using CapriKit.AssetPipeline; +using CapriKit.IO; +using CapriKit.IO.Buffers; +using System.Buffers; + +namespace CapriKit.Tests.AssetPipeline; + +internal class AssetEncoderTests +{ + [Test] + public async Task Encode() + { + var fileSystem = new InMemoryFileSystem(); + await fileSystem.WriteAllText("hello.txt", "héllo"); + var transcoder = new DummyTranscoder(); + var id = new AssetId("Main", "hello.txt"); + + await AssetEncoder.Encode(id, fileSystem, transcoder); + var bytes = await fileSystem.ReadAllBytes("hello.txt.cka"); + + var reader = new SequenceReader(new ReadOnlySequence(bytes)); + var encoderId = reader.ReadGuid(); + var encoderVersion = reader.ReadInt32(); + var payloadLength = reader.ReadInt32(); + reader.Advance(payloadLength); + var dependencyCount = reader.ReadInt32(); + var dependency = reader.ReadString(); + var end = reader.End; + + await Assert.That(encoderId).IsEqualTo(transcoder.Id); + await Assert.That(encoderVersion).IsEqualTo(transcoder.Version); + await Assert.That(dependencyCount).IsEqualTo(1); + await Assert.That(dependency).IsEqualTo("hello.txt"); + await Assert.That(end).IsTrue(); + } +} diff --git a/source/CapriKit.Tests/AssetPipeline/AssetTranscoderTests.cs b/source/CapriKit.Tests/AssetPipeline/AssetTranscoderTests.cs deleted file mode 100644 index bc152cc..0000000 --- a/source/CapriKit.Tests/AssetPipeline/AssetTranscoderTests.cs +++ /dev/null @@ -1,82 +0,0 @@ -using CapriKit.AssetPipeline; -using CapriKit.IO; -using CapriKit.IO.Buffers; -using System.Buffers; - -namespace CapriKit.Tests.AssetPipeline; - -internal class AssetTranscoderTests -{ - - [Test] - public async Task Decode() - { - var fileSystem = new InMemoryFileSystem(); - await fileSystem.WriteAllText("hello.txt", "héllo"); - var transcoder = new DummyTranscoder(); - var id = new AssetId("Main", "hello.txt"); - - await AssetTranscoder.Encode(id, fileSystem, transcoder); - var envelope = await AssetTranscoder.Decode(id, fileSystem, transcoder); - - FilePath expectedDependency = "hello.txt"; - await Assert.That(envelope.Asset).IsEqualTo("HÉLLO"); - await Assert.That(envelope.Dependencies.Count).IsEqualTo(1); - await Assert.That(envelope.Dependencies.First()).IsEqualTo(expectedDependency); - } - - [Test] - public async Task Encode() - { - var fileSystem = new InMemoryFileSystem(); - await fileSystem.WriteAllText("hello.txt", "héllo"); - var transcoder = new DummyTranscoder(); - var id = new AssetId("Main", "hello.txt"); - - await AssetTranscoder.Encode(id, fileSystem, transcoder); - var bytes = await fileSystem.ReadAllBytes("hello.txt.cka"); - - // SequenceReader is a ref struct, so all reading happens before the first - // await and only plain locals cross into the assertions - var reader = new SequenceReader(new ReadOnlySequence(bytes)); - var encoderId = reader.ReadGuid(); - var encoderVersion = reader.ReadInt32(); - var payloadLength = reader.ReadInt32(); - reader.Advance(payloadLength); - var dependencyCount = reader.ReadInt32(); - var dependency = reader.ReadString(); - var end = reader.End; - - await Assert.That(encoderId).IsEqualTo(transcoder.Id); - await Assert.That(encoderVersion).IsEqualTo(transcoder.Version); - await Assert.That(dependencyCount).IsEqualTo(1); - await Assert.That(dependency).IsEqualTo("hello.txt"); - await Assert.That(end).IsTrue(); - } - - /// - /// Uppercases a text file - /// - private sealed class DummyTranscoder : IAssetEncoder, IAssetDecoder - { - public IReadOnlySet SupportedExtensions { get; } = new HashSet([".txt"]); - public Guid Id => Guid.Parse("{B87F41E3-6C33-46E4-802A-3E1E82800E7A}"); - public int Version => 1; - - public async Task Encode(AssetId id, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer) - { - var text = await fileSystem.ReadAllText(id.Path); - writer.Write(text.ToUpperInvariant()); - } - - public string Decode(AssetId id, ref SequenceReader reader) - { - return reader.ReadString(); - } - - public void HotSwap(string instance, string replacement) - { - throw new NotImplementedException(); - } - } -} diff --git a/source/CapriKit.Tests/AssetPipeline/DummyTranscoder.cs b/source/CapriKit.Tests/AssetPipeline/DummyTranscoder.cs new file mode 100644 index 0000000..c29a769 --- /dev/null +++ b/source/CapriKit.Tests/AssetPipeline/DummyTranscoder.cs @@ -0,0 +1,32 @@ +using CapriKit.AssetPipeline; +using CapriKit.IO; +using CapriKit.IO.Buffers; +using System.Buffers; + +namespace CapriKit.Tests.AssetPipeline; + +/// +/// Uppercases a text file +/// +internal sealed class DummyTranscoder : IAssetEncoder, IAssetDecoder +{ + public IReadOnlySet SupportedExtensions { get; } = new HashSet([".txt"]); + public Guid Id => Guid.Parse("{B87F41E3-6C33-46E4-802A-3E1E82800E7A}"); + public int Version => 1; + + public async Task Encode(AssetId id, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer) + { + var text = await fileSystem.ReadAllText(id.Path); + writer.Write(text.ToUpperInvariant()); + } + + public string Decode(AssetId id, ref SequenceReader reader) + { + return reader.ReadString(); + } + + public void HotSwap(string instance, string replacement) + { + throw new NotImplementedException(); + } +} From c177ba516525ccb6e507ffb2ddd216b03913479c Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Tue, 14 Jul 2026 21:16:13 +0200 Subject: [PATCH 05/53] Move dependency towards AssetPipeline project --- .../CapriKit.AssetPipeline.csproj | 5 ++ .../Shaders/ShaderTranscoder.cs | 28 ++++++++ .../Shaders/VertexShaderTranscoder.cs | 32 +++++++++ .../Assets/ITextureEncoder.cs | 10 --- .../Assets/VertexShaderTranscoder.cs | 44 ------------- .../CapriKit.DirectX11.csproj | 1 - .../Resources/Shaders/ComputeShader.cs | 65 +++++++++++++++---- .../Resources/Shaders/PixelShader.cs | 18 +++-- .../Resources/Shaders/ShaderCompiler.cs | 48 +++++++++----- .../Resources/Shaders/VertexShader.cs | 32 ++++++--- 10 files changed, 186 insertions(+), 97 deletions(-) create mode 100644 source/CapriKit.AssetPipeline/Shaders/ShaderTranscoder.cs create mode 100644 source/CapriKit.AssetPipeline/Shaders/VertexShaderTranscoder.cs delete mode 100644 source/CapriKit.DirectX11/Assets/ITextureEncoder.cs delete mode 100644 source/CapriKit.DirectX11/Assets/VertexShaderTranscoder.cs diff --git a/source/CapriKit.AssetPipeline/CapriKit.AssetPipeline.csproj b/source/CapriKit.AssetPipeline/CapriKit.AssetPipeline.csproj index 4e4824a..3728e07 100644 --- a/source/CapriKit.AssetPipeline/CapriKit.AssetPipeline.csproj +++ b/source/CapriKit.AssetPipeline/CapriKit.AssetPipeline.csproj @@ -1,5 +1,10 @@  + + $(TargetFramework)-windows + + + diff --git a/source/CapriKit.AssetPipeline/Shaders/ShaderTranscoder.cs b/source/CapriKit.AssetPipeline/Shaders/ShaderTranscoder.cs new file mode 100644 index 0000000..9789d1d --- /dev/null +++ b/source/CapriKit.AssetPipeline/Shaders/ShaderTranscoder.cs @@ -0,0 +1,28 @@ +using CapriKit.DirectX11.Resources.Shaders; +using CapriKit.IO.Buffers; +using System.Buffers; + +namespace CapriKit.AssetPipeline.Shaders; + +internal static class ShaderTranscoder +{ + public static IReadOnlySet SupportedExtensions { get; } = new HashSet([".hlsl"]); + + public static void WriteCommon(ShaderByteCode shader, IBufferWriter writer) + { + writer.Write(shader.EntryPoint); + writer.Write(shader.Name); + writer.Write(shader.Bytes.Length); + writer.Write(shader.Bytes); + } + + public static ShaderByteCode ReadCommon(ref SequenceReader reader) + { + var entryPoint = reader.ReadString(); + var name = reader.ReadString(); + var length = reader.ReadInt32(); + var bytes = reader.ReadBytes(length); + + return new ShaderByteCode(bytes, entryPoint, name); + } +} diff --git a/source/CapriKit.AssetPipeline/Shaders/VertexShaderTranscoder.cs b/source/CapriKit.AssetPipeline/Shaders/VertexShaderTranscoder.cs new file mode 100644 index 0000000..59a3162 --- /dev/null +++ b/source/CapriKit.AssetPipeline/Shaders/VertexShaderTranscoder.cs @@ -0,0 +1,32 @@ +using CapriKit.DirectX11; +using CapriKit.DirectX11.Resources.Shaders; +using CapriKit.IO; +using System.Buffers; + +namespace CapriKit.AssetPipeline.Shaders; + +internal sealed class VertexShaderTranscoder(Device device) : IAssetEncoder, IAssetDecoder +{ + public IReadOnlySet SupportedExtensions => ShaderTranscoder.SupportedExtensions; + public Guid Id => Guid.Parse("{CA3CB37D-9880-4B61-AB09-EBC17E7533E6}"); + public int Version => 1; + + public async Task Encode(AssetId id, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer) + { + var source = await fileSystem.ReadAllText(id.Path); + var includePath = id.Path.Directory; + var bytes = ShaderCompiler.CompileVertexShader(fileSystem, includePath, source, id.Key, id.ToString()); + ShaderTranscoder.WriteCommon(bytes.Common, writer); + } + + public IVertexShader Decode(AssetId id, ref SequenceReader reader) + { + var common = ShaderTranscoder.ReadCommon(ref reader); + return ShaderCompiler.CreateVertexShader(new VertexShaderByteCode(common), device); + } + + public void HotSwap(IVertexShader instance, IVertexShader replacement) + { + instance.HotSwap(replacement); + } +} diff --git a/source/CapriKit.DirectX11/Assets/ITextureEncoder.cs b/source/CapriKit.DirectX11/Assets/ITextureEncoder.cs deleted file mode 100644 index 83694a0..0000000 --- a/source/CapriKit.DirectX11/Assets/ITextureEncoder.cs +++ /dev/null @@ -1,10 +0,0 @@ -namespace CapriKit.DirectX11.Assets; - -public interface ITextureEncoder -{ -} - -public interface ITextureDecoder -{ - -} diff --git a/source/CapriKit.DirectX11/Assets/VertexShaderTranscoder.cs b/source/CapriKit.DirectX11/Assets/VertexShaderTranscoder.cs deleted file mode 100644 index 6db4efb..0000000 --- a/source/CapriKit.DirectX11/Assets/VertexShaderTranscoder.cs +++ /dev/null @@ -1,44 +0,0 @@ -using CapriKit.AssetPipeline; -using CapriKit.DirectX11.Resources.Shaders; -using CapriKit.IO; -using CapriKit.IO.Buffers; -using System.Buffers; - -namespace CapriKit.DirectX11.Assets; - -public sealed class VertexShaderTranscoder(Device device) : IAssetEncoder, IAssetDecoder -{ - public IReadOnlySet SupportedExtensions { get; } = new HashSet([".hlsl"]); - public Guid Id => Guid.Parse("{CA3CB37D-9880-4B61-AB09-EBC17E7533E6}"); - public int Version => 1; - - public async Task Encode(AssetId id, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer) - { - var source = await fileSystem.ReadAllText(id.Path); - var includePath = id.Path.Directory; - var bytes = ShaderCompiler.CompileVertexShader(fileSystem, includePath, source, id.Key, id.ToString()); - - writer.Write(bytes.EntryPoint); - writer.Write(bytes.Name); - writer.Write(bytes.Bytes.Length); - writer.Write(bytes.Bytes); - } - - public IVertexShader Decode(AssetId id, ref SequenceReader reader) - { - var entryPoint = reader.ReadString(); - var name = reader.ReadString(); - var length = reader.ReadInt32(); - var bytes = reader.ReadBytes(length); - - var byteCode = new VertexShaderByteCode(bytes, entryPoint, name); - return ShaderCompiler.CreateVertexShader(byteCode, device); - } - - public void HotSwap(IVertexShader instance, IVertexShader replacement) - { - var original = instance.ID3D11VertexShader; - instance.ID3D11VertexShader = replacement.ID3D11VertexShader; - original.Dispose(); - } -} diff --git a/source/CapriKit.DirectX11/CapriKit.DirectX11.csproj b/source/CapriKit.DirectX11/CapriKit.DirectX11.csproj index 8e565dc..d4db051 100644 --- a/source/CapriKit.DirectX11/CapriKit.DirectX11.csproj +++ b/source/CapriKit.DirectX11/CapriKit.DirectX11.csproj @@ -8,7 +8,6 @@ - diff --git a/source/CapriKit.DirectX11/Resources/Shaders/ComputeShader.cs b/source/CapriKit.DirectX11/Resources/Shaders/ComputeShader.cs index e615f9b..581c23a 100644 --- a/source/CapriKit.DirectX11/Resources/Shaders/ComputeShader.cs +++ b/source/CapriKit.DirectX11/Resources/Shaders/ComputeShader.cs @@ -4,7 +4,22 @@ namespace CapriKit.DirectX11.Resources.Shaders; public interface IComputeShader : IDisposable { - internal ID3D11ComputeShader ID3D11ComputeShader { get; } + internal uint NumThreadsX { get; set; } + internal uint NumThreadsY { get; set; } + internal uint NumThreadsZ { get; set; } + internal ID3D11ComputeShader ID3D11ComputeShader { get; set; } + + + public void HotSwap(IComputeShader replacement) + { + NumThreadsX = replacement.NumThreadsX; + NumThreadsY = replacement.NumThreadsY; + NumThreadsZ = replacement.NumThreadsZ; + + var oldShader = ID3D11ComputeShader; + ID3D11ComputeShader = replacement.ID3D11ComputeShader; + oldShader.Dispose(); + } /// /// The shader kernel defines how big the groups for each dimension are. @@ -16,27 +31,49 @@ public interface IComputeShader : IDisposable internal sealed class ComputeShader : IComputeShader { - private readonly uint NumThreadsX; - private readonly uint NumThreadsY; - private readonly uint NumThreadsZ; - private readonly ID3D11ComputeShader Shader; + private uint numThreadsX; + private uint numThreadsY; + private uint numThreadsZ; + private ID3D11ComputeShader shader; internal ComputeShader(ID3D11ComputeShader shader, uint numThreadsX, uint numThreadsY, uint numThreadsZ) { - Shader = shader; - NumThreadsX = numThreadsX; - NumThreadsY = numThreadsY; - NumThreadsZ = numThreadsZ; + this.shader = shader; + this.numThreadsX = numThreadsX; + this.numThreadsY = numThreadsY; + this.numThreadsZ = numThreadsZ; + } + + ID3D11ComputeShader IComputeShader.ID3D11ComputeShader + { + get { return shader; } + set { shader = value; } } - ID3D11ComputeShader IComputeShader.ID3D11ComputeShader => Shader; + uint IComputeShader.NumThreadsX + { + get { return numThreadsX; } + set { numThreadsX = value; } + } + + uint IComputeShader.NumThreadsY + { + get { return numThreadsY; } + set { numThreadsY = value; } + } + + uint IComputeShader.NumThreadsZ + { + get { return numThreadsZ; } + set { numThreadsZ = value; } + } /// public (uint X, uint Y, uint Z) GetDispatchSize(uint dimX, uint dimY, uint dimZ) { - var x = GetDispatchSize(NumThreadsX, dimX); - var y = GetDispatchSize(NumThreadsY, dimY); - var z = GetDispatchSize(NumThreadsZ, dimZ); + var x = GetDispatchSize(numThreadsX, dimX); + var y = GetDispatchSize(numThreadsY, dimY); + var z = GetDispatchSize(numThreadsZ, dimZ); return (x, y, z); } @@ -48,6 +85,6 @@ private static uint GetDispatchSize(uint numThreads, uint dim) public void Dispose() { - Shader.Dispose(); + shader.Dispose(); } } diff --git a/source/CapriKit.DirectX11/Resources/Shaders/PixelShader.cs b/source/CapriKit.DirectX11/Resources/Shaders/PixelShader.cs index f0146e4..8eae3f1 100644 --- a/source/CapriKit.DirectX11/Resources/Shaders/PixelShader.cs +++ b/source/CapriKit.DirectX11/Resources/Shaders/PixelShader.cs @@ -4,20 +4,30 @@ namespace CapriKit.DirectX11.Resources.Shaders; public interface IPixelShader : IDisposable { - internal ID3D11PixelShader ID3D11PixelShader { get; } + internal ID3D11PixelShader ID3D11PixelShader { get; set; } + + public void HotSwap(IPixelShader replacement) + { + var oldShader = ID3D11PixelShader; + ID3D11PixelShader = replacement.ID3D11PixelShader; + oldShader.Dispose(); + } } internal sealed class PixelShader : IPixelShader { - private readonly ID3D11PixelShader Shader; + private ID3D11PixelShader Shader; internal PixelShader(ID3D11PixelShader shader) { Shader = shader; } - ID3D11PixelShader IPixelShader.ID3D11PixelShader => Shader; - + ID3D11PixelShader IPixelShader.ID3D11PixelShader + { + get { return Shader; } + set { Shader = value; } + } public void Dispose() { Shader.Dispose(); diff --git a/source/CapriKit.DirectX11/Resources/Shaders/ShaderCompiler.cs b/source/CapriKit.DirectX11/Resources/Shaders/ShaderCompiler.cs index a400951..5ea5b9f 100644 --- a/source/CapriKit.DirectX11/Resources/Shaders/ShaderCompiler.cs +++ b/source/CapriKit.DirectX11/Resources/Shaders/ShaderCompiler.cs @@ -5,13 +5,29 @@ namespace CapriKit.DirectX11.Resources.Shaders; -public abstract record ShaderByteCode(byte[] Bytes, string EntryPoint, string Name); +public record ShaderByteCode(byte[] Bytes, string EntryPoint, string Name); -public sealed record VertexShaderByteCode(byte[] Bytes, string EntryPoint, string Name) : ShaderByteCode(Bytes, EntryPoint, Name); +public sealed record VertexShaderByteCode(ShaderByteCode Common) +{ + public byte[] Bytes => Common.Bytes; + public string EntryPoint => Common.EntryPoint; + public string Name => Common.Name; +} -public sealed record PixelShaderByteCode(byte[] Bytes, string EntryPoint, string Name) : ShaderByteCode(Bytes, EntryPoint, Name); +public sealed record PixelShaderByteCode(ShaderByteCode Common) +{ + public byte[] Bytes => Common.Bytes; + public string EntryPoint => Common.EntryPoint; + public string Name => Common.Name; +} -public sealed record ComputeShaderByteCode(byte[] Bytes, uint NumThreadsX, uint NumThreadsY, uint NumThreadsZ, string EntryPoint, string Name) : ShaderByteCode(Bytes, EntryPoint, Name); + +public sealed record ComputeShaderByteCode(ShaderByteCode Common, uint NumThreadsX, uint NumThreadsY, uint NumThreadsZ) +{ + public byte[] Bytes => Common.Bytes; + public string EntryPoint => Common.EntryPoint; + public string Name => Common.Name; +} public static class ShaderCompiler { @@ -21,14 +37,14 @@ public static class ShaderCompiler public static IVertexShader CompileVertexShader(IReadOnlyVirtualFileSystem fileSystem, DirectoryPath includePath, Device device, string source, string entryPoint, string name) { - var byteCode = CompileVertexShader(fileSystem, includePath, source, entryPoint, name); - return CreateVertexShader(byteCode, device); + var bytes = CompileVertexShader(fileSystem, includePath, source, entryPoint, name); + return CreateVertexShader(bytes, device); } public static VertexShaderByteCode CompileVertexShader(IReadOnlyVirtualFileSystem fileSystem, DirectoryPath includePath, string source, string entryPoint, string name) { - var bytes = Compile(fileSystem, includePath, source, entryPoint, name, VERTEX_SHADER_PROFILE); - return new VertexShaderByteCode(bytes, entryPoint, name); + var byteCode = Compile(fileSystem, includePath, source, entryPoint, name, VERTEX_SHADER_PROFILE); + return new VertexShaderByteCode(byteCode); } public static IVertexShader CreateVertexShader(VertexShaderByteCode byteCode, Device device) @@ -47,7 +63,7 @@ public static IPixelShader CompilePixelShader(IReadOnlyVirtualFileSystem fileSys public static PixelShaderByteCode CompilePixelShader(IReadOnlyVirtualFileSystem fileSystem, DirectoryPath includePath, string source, string entryPoint, string name) { var bytes = Compile(fileSystem, includePath, source, entryPoint, name, PIXEL_SHADER_PROFILE); - return new PixelShaderByteCode(bytes, entryPoint, name); + return new PixelShaderByteCode(bytes); } public static IPixelShader CreatePixelShader(PixelShaderByteCode byteCode, Device device) { @@ -58,15 +74,15 @@ public static IPixelShader CreatePixelShader(PixelShaderByteCode byteCode, Devic public static IComputeShader CompileComputeShader(IReadOnlyVirtualFileSystem fileSystem, DirectoryPath includePath, Device device, string source, string entryPoint, string name) { - var byteCode = CompileComputeShader(fileSystem, includePath, source, entryPoint, name); - return CreateComputeShader(byteCode, device); + var common = CompileComputeShader(fileSystem, includePath, source, entryPoint, name); + return CreateComputeShader(common, device); } public static ComputeShaderByteCode CompileComputeShader(IReadOnlyVirtualFileSystem fileSystem, DirectoryPath includePath, string source, string entryPoint, string name) { - var bytes = Compile(fileSystem, includePath, source, entryPoint, name, COMPUTE_SHADER_PROFILE); - var (x, y, z) = QueryNumThreads(bytes, name); - return new ComputeShaderByteCode(bytes, x, y, z, entryPoint, name); + var common = Compile(fileSystem, includePath, source, entryPoint, name, COMPUTE_SHADER_PROFILE); + var (x, y, z) = QueryNumThreads(common.Bytes, name); + return new ComputeShaderByteCode(common, x, y, z); } public static IComputeShader CreateComputeShader(ComputeShaderByteCode byteCode, Device device) @@ -76,7 +92,7 @@ public static IComputeShader CreateComputeShader(ComputeShaderByteCode byteCode, return new ComputeShader(shader, byteCode.NumThreadsX, byteCode.NumThreadsY, byteCode.NumThreadsZ); } - private static byte[] Compile(IReadOnlyVirtualFileSystem fileSystem, DirectoryPath includePath, string source, string entryPoint, string name, string profile) + private static ShaderByteCode Compile(IReadOnlyVirtualFileSystem fileSystem, DirectoryPath includePath, string source, string entryPoint, string name, string profile) { using var includeResolver = new ShaderIncludeResolver(fileSystem, includePath); @@ -92,7 +108,7 @@ private static byte[] Compile(IReadOnlyVirtualFileSystem fileSystem, DirectoryPa var bytes = blob.AsBytes(); blob.Dispose(); - return bytes; + return new ShaderByteCode(bytes, entryPoint, name); } /// diff --git a/source/CapriKit.DirectX11/Resources/Shaders/VertexShader.cs b/source/CapriKit.DirectX11/Resources/Shaders/VertexShader.cs index a26b072..31e9fa4 100644 --- a/source/CapriKit.DirectX11/Resources/Shaders/VertexShader.cs +++ b/source/CapriKit.DirectX11/Resources/Shaders/VertexShader.cs @@ -4,36 +4,52 @@ namespace CapriKit.DirectX11.Resources.Shaders; public interface IVertexShader : IDisposable { + internal byte[] Blob { get; set; } internal ID3D11VertexShader ID3D11VertexShader { get; set; } IInputLayout CreateInputLayout(Device device, InputElementDescription[] elements); + + public void HotSwap(IVertexShader replacement) + { + Blob = replacement.Blob; + + var oldShader = ID3D11VertexShader; + ID3D11VertexShader = replacement.ID3D11VertexShader; + oldShader.Dispose(); + } } internal sealed class VertexShader : IVertexShader { - private readonly byte[] Blob; - private ID3D11VertexShader Shader; + private byte[] blob; + private ID3D11VertexShader shader; internal VertexShader(byte[] blob, ID3D11VertexShader shader) { - Blob = blob; - Shader = shader; + this.blob = blob; + this.shader = shader; + } + + byte[] IVertexShader.Blob + { + get { return blob; } + set { blob = value; } } ID3D11VertexShader IVertexShader.ID3D11VertexShader { - get { return Shader; } - set { Shader = value; } + get { return shader; } + set { shader = value; } } public IInputLayout CreateInputLayout(Device device, InputElementDescription[] elements) { - var inputLayout = device.ID3D11Device.CreateInputLayout(elements, Blob); + var inputLayout = device.ID3D11Device.CreateInputLayout(elements, blob); return new InputLayout(inputLayout); } public void Dispose() { - Shader.Dispose(); + shader.Dispose(); } } From 797a1d045428f9487faec5982bc3e01fc494dd9d Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Tue, 14 Jul 2026 21:34:49 +0200 Subject: [PATCH 06/53] add idea --- .../CapriKit.AssetPipeline/Shaders/VertexShaderTranscoder.cs | 3 +++ 1 file changed, 3 insertions(+) diff --git a/source/CapriKit.AssetPipeline/Shaders/VertexShaderTranscoder.cs b/source/CapriKit.AssetPipeline/Shaders/VertexShaderTranscoder.cs index 59a3162..cddeece 100644 --- a/source/CapriKit.AssetPipeline/Shaders/VertexShaderTranscoder.cs +++ b/source/CapriKit.AssetPipeline/Shaders/VertexShaderTranscoder.cs @@ -5,6 +5,9 @@ namespace CapriKit.AssetPipeline.Shaders; +// TODO: I am not happy that this code is here, but I also do not want to put it in CapriKit.DirectX11 +// maybe I can make a CapriKit.AssetPipeline.DirectX11 project that has all the DirectX11 specific bits? +// (shaders, textures) internal sealed class VertexShaderTranscoder(Device device) : IAssetEncoder, IAssetDecoder { public IReadOnlySet SupportedExtensions => ShaderTranscoder.SupportedExtensions; From 32ef4b8cd93f818d9d852a8ccedc7c251465a120 Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Thu, 16 Jul 2026 17:00:01 +0200 Subject: [PATCH 07/53] Split out assets that need DirectX11 and generic assets --- CapriKit.slnx | 3 ++ .../CapriKit.AssetPipeline.DirectX11.csproj | 11 ++++++ .../Shaders/ShaderTranscoder.cs | 5 +-- .../Shaders/VertexShaderTranscoder.cs | 7 ++-- source/CapriKit.AssetPipeline/Asset.cs | 5 +++ source/CapriKit.AssetPipeline/AssetDecoder.cs | 20 +++++++++-- source/CapriKit.AssetPipeline/AssetEncoder.cs | 9 +++++ source/CapriKit.AssetPipeline/AssetId.cs | 5 +++ source/CapriKit.AssetPipeline/AssetManager.cs | 30 ++++++++++++++++ .../CapriKit.AssetPipeline.csproj | 5 --- source/CapriKit.AssetPipeline/Envelope.cs | 5 --- .../CapriKit.AssetPipeline/IAssetCompiler.cs | 34 ------------------- .../AssetPipeline/AssetDecoderTests.cs | 2 +- source/CapriKit.Tests/CapriKit.Tests.csproj | 3 -- .../DirectX11/Buffers/GenericBufferTests.cs | 2 +- 15 files changed, 88 insertions(+), 58 deletions(-) create mode 100644 source/CapriKit.AssetPipeline.DirectX11/CapriKit.AssetPipeline.DirectX11.csproj rename source/{CapriKit.AssetPipeline => CapriKit.AssetPipeline.DirectX11}/Shaders/ShaderTranscoder.cs (82%) rename source/{CapriKit.AssetPipeline => CapriKit.AssetPipeline.DirectX11}/Shaders/VertexShaderTranscoder.cs (72%) create mode 100644 source/CapriKit.AssetPipeline/Asset.cs create mode 100644 source/CapriKit.AssetPipeline/AssetId.cs create mode 100644 source/CapriKit.AssetPipeline/AssetManager.cs delete mode 100644 source/CapriKit.AssetPipeline/Envelope.cs delete mode 100644 source/CapriKit.AssetPipeline/IAssetCompiler.cs diff --git a/CapriKit.slnx b/CapriKit.slnx index 9e27b76..4c79f7b 100644 --- a/CapriKit.slnx +++ b/CapriKit.slnx @@ -32,6 +32,9 @@ + + + diff --git a/source/CapriKit.AssetPipeline.DirectX11/CapriKit.AssetPipeline.DirectX11.csproj b/source/CapriKit.AssetPipeline.DirectX11/CapriKit.AssetPipeline.DirectX11.csproj new file mode 100644 index 0000000..c02ac0d --- /dev/null +++ b/source/CapriKit.AssetPipeline.DirectX11/CapriKit.AssetPipeline.DirectX11.csproj @@ -0,0 +1,11 @@ + + + $(TargetFramework)-windows + + + + + + + + diff --git a/source/CapriKit.AssetPipeline/Shaders/ShaderTranscoder.cs b/source/CapriKit.AssetPipeline.DirectX11/Shaders/ShaderTranscoder.cs similarity index 82% rename from source/CapriKit.AssetPipeline/Shaders/ShaderTranscoder.cs rename to source/CapriKit.AssetPipeline.DirectX11/Shaders/ShaderTranscoder.cs index 9789d1d..2891395 100644 --- a/source/CapriKit.AssetPipeline/Shaders/ShaderTranscoder.cs +++ b/source/CapriKit.AssetPipeline.DirectX11/Shaders/ShaderTranscoder.cs @@ -1,12 +1,13 @@ using CapriKit.DirectX11.Resources.Shaders; using CapriKit.IO.Buffers; using System.Buffers; +using System.Collections.Frozen; -namespace CapriKit.AssetPipeline.Shaders; +namespace CapriKit.AssetPipeline.DirectX11.Shaders; internal static class ShaderTranscoder { - public static IReadOnlySet SupportedExtensions { get; } = new HashSet([".hlsl"]); + public static IReadOnlySet SupportedExtensions { get; } = new HashSet([".hlsl"]).ToFrozenSet(); public static void WriteCommon(ShaderByteCode shader, IBufferWriter writer) { diff --git a/source/CapriKit.AssetPipeline/Shaders/VertexShaderTranscoder.cs b/source/CapriKit.AssetPipeline.DirectX11/Shaders/VertexShaderTranscoder.cs similarity index 72% rename from source/CapriKit.AssetPipeline/Shaders/VertexShaderTranscoder.cs rename to source/CapriKit.AssetPipeline.DirectX11/Shaders/VertexShaderTranscoder.cs index cddeece..1aef183 100644 --- a/source/CapriKit.AssetPipeline/Shaders/VertexShaderTranscoder.cs +++ b/source/CapriKit.AssetPipeline.DirectX11/Shaders/VertexShaderTranscoder.cs @@ -3,12 +3,9 @@ using CapriKit.IO; using System.Buffers; -namespace CapriKit.AssetPipeline.Shaders; +namespace CapriKit.AssetPipeline.DirectX11.Shaders; -// TODO: I am not happy that this code is here, but I also do not want to put it in CapriKit.DirectX11 -// maybe I can make a CapriKit.AssetPipeline.DirectX11 project that has all the DirectX11 specific bits? -// (shaders, textures) -internal sealed class VertexShaderTranscoder(Device device) : IAssetEncoder, IAssetDecoder +public sealed class VertexShaderTranscoder(Device device) : IAssetEncoder, IAssetDecoder { public IReadOnlySet SupportedExtensions => ShaderTranscoder.SupportedExtensions; public Guid Id => Guid.Parse("{CA3CB37D-9880-4B61-AB09-EBC17E7533E6}"); diff --git a/source/CapriKit.AssetPipeline/Asset.cs b/source/CapriKit.AssetPipeline/Asset.cs new file mode 100644 index 0000000..5cbcfbb --- /dev/null +++ b/source/CapriKit.AssetPipeline/Asset.cs @@ -0,0 +1,5 @@ +using CapriKit.IO; + +namespace CapriKit.AssetPipeline; + +internal sealed record Asset(T Value, IReadOnlySet Dependencies); diff --git a/source/CapriKit.AssetPipeline/AssetDecoder.cs b/source/CapriKit.AssetPipeline/AssetDecoder.cs index 40eda84..952fed5 100644 --- a/source/CapriKit.AssetPipeline/AssetDecoder.cs +++ b/source/CapriKit.AssetPipeline/AssetDecoder.cs @@ -6,9 +6,25 @@ namespace CapriKit.AssetPipeline; +public interface IAssetDecoder +{ + Guid Id { get; } + int Version { get; } + +} + +public interface IAssetDecoder : IAssetDecoder +{ + // Synchronous by design: the envelope owns all file IO and hands the decoder an + // in-memory payload. The reader's buffer is only valid for the duration of the call, + // decoders must copy out anything they want to keep. + TAsset Decode(AssetId id, ref SequenceReader reader); + void HotSwap(TAsset instance, TAsset replacement); +} + internal static class AssetDecoder { - public static async Task> Decode(AssetId id, IVirtualFileSystem fileSystem, IAssetDecoder decoder) + public static async Task> Decode(AssetId id, IVirtualFileSystem fileSystem, IAssetDecoder decoder) { var inputPath = ToEncodedFilePath(id.Path); ThrowOnFileNotFound(inputPath, fileSystem); @@ -19,7 +35,7 @@ public static async Task> Decode(AssetId id, IVirtualFileSystem f var asset = await ReadPayload(input, payloadLength, decoder, id); var dependencies = await ReadDependencies(input); - return new Envelope(asset, dependencies); + return new Asset(asset, dependencies); } private static async Task ReadHeader(Stream input, IAssetDecoder decoder, FilePath path) diff --git a/source/CapriKit.AssetPipeline/AssetEncoder.cs b/source/CapriKit.AssetPipeline/AssetEncoder.cs index 59fb412..735ec54 100644 --- a/source/CapriKit.AssetPipeline/AssetEncoder.cs +++ b/source/CapriKit.AssetPipeline/AssetEncoder.cs @@ -6,6 +6,15 @@ namespace CapriKit.AssetPipeline; +public interface IAssetEncoder +{ + IReadOnlySet SupportedExtensions { get; } + Guid Id { get; } + int Version { get; } + + Task Encode(AssetId id, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer); +} + internal sealed class AssetEncoder { public static async Task Encode(AssetId id, IVirtualFileSystem fileSystem, IAssetEncoder encoder) diff --git a/source/CapriKit.AssetPipeline/AssetId.cs b/source/CapriKit.AssetPipeline/AssetId.cs new file mode 100644 index 0000000..d41a2ec --- /dev/null +++ b/source/CapriKit.AssetPipeline/AssetId.cs @@ -0,0 +1,5 @@ +using CapriKit.IO; + +namespace CapriKit.AssetPipeline; + +public record AssetId(string Key, FilePath Path); diff --git a/source/CapriKit.AssetPipeline/AssetManager.cs b/source/CapriKit.AssetPipeline/AssetManager.cs new file mode 100644 index 0000000..5349312 --- /dev/null +++ b/source/CapriKit.AssetPipeline/AssetManager.cs @@ -0,0 +1,30 @@ +namespace CapriKit.AssetPipeline; + +public class AssetManager +{ + + private readonly Dictionary Encoders = []; + private readonly Dictionary Decoders = []; + + public void RegisterTranscoder(IAssetEncoder encoder, IAssetDecoder decoder) + { + var typeKey = typeof(T); + Encoders[typeKey] = encoder; + Decoders[typeKey] = decoder; + } + + public T Decode(AssetId id) + { + var typeKey = typeof(T); + var encoder = Encoders[typeKey]; + var decoder = Decoders[typeKey]; + + var extension = new string(id.Path.Extension); + if (encoder.SupportedExtensions.Contains(extension)) + { + + } + + throw new NotImplementedException(); + } +} diff --git a/source/CapriKit.AssetPipeline/CapriKit.AssetPipeline.csproj b/source/CapriKit.AssetPipeline/CapriKit.AssetPipeline.csproj index 3728e07..4e4824a 100644 --- a/source/CapriKit.AssetPipeline/CapriKit.AssetPipeline.csproj +++ b/source/CapriKit.AssetPipeline/CapriKit.AssetPipeline.csproj @@ -1,10 +1,5 @@  - - $(TargetFramework)-windows - - - diff --git a/source/CapriKit.AssetPipeline/Envelope.cs b/source/CapriKit.AssetPipeline/Envelope.cs deleted file mode 100644 index 3cf8129..0000000 --- a/source/CapriKit.AssetPipeline/Envelope.cs +++ /dev/null @@ -1,5 +0,0 @@ -using CapriKit.IO; - -namespace CapriKit.AssetPipeline; - -internal sealed record Envelope(T Asset, IReadOnlySet Dependencies); diff --git a/source/CapriKit.AssetPipeline/IAssetCompiler.cs b/source/CapriKit.AssetPipeline/IAssetCompiler.cs deleted file mode 100644 index 6e8fb7b..0000000 --- a/source/CapriKit.AssetPipeline/IAssetCompiler.cs +++ /dev/null @@ -1,34 +0,0 @@ -using CapriKit.IO; -using System.Buffers; - -namespace CapriKit.AssetPipeline; - -public record AssetId(string Key, FilePath Path); - -public interface IAssetEncoder -{ - IReadOnlySet SupportedExtensions { get; } - Guid Id { get; } - int Version { get; } - - Task Encode(AssetId id, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer); -} - -// TODO: images need settings and we could even say shaders do (the entry point) if we do not want to -// rely on AssetId.Key -public interface IAssetEncoder : IAssetEncoder -{ - Task Encode(AssetId id, TSettings settings, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer); -} - -public interface IAssetDecoder -{ - Guid Id { get; } - int Version { get; } - - // Synchronous by design: the envelope owns all file IO and hands the decoder an - // in-memory payload. The reader's buffer is only valid for the duration of the call, - // decoders must copy out anything they want to keep. - TAsset Decode(AssetId id, ref SequenceReader reader); - void HotSwap(TAsset instance, TAsset replacement); -} diff --git a/source/CapriKit.Tests/AssetPipeline/AssetDecoderTests.cs b/source/CapriKit.Tests/AssetPipeline/AssetDecoderTests.cs index 786d2f7..dea0922 100644 --- a/source/CapriKit.Tests/AssetPipeline/AssetDecoderTests.cs +++ b/source/CapriKit.Tests/AssetPipeline/AssetDecoderTests.cs @@ -18,7 +18,7 @@ public async Task Decode() var envelope = await AssetDecoder.Decode(id, fileSystem, transcoder); FilePath expectedDependency = "hello.txt"; - await Assert.That(envelope.Asset).IsEqualTo("HÉLLO"); + await Assert.That(envelope.Value).IsEqualTo("HÉLLO"); await Assert.That(envelope.Dependencies.Count).IsEqualTo(1); await Assert.That(envelope.Dependencies.First()).IsEqualTo(expectedDependency); } diff --git a/source/CapriKit.Tests/CapriKit.Tests.csproj b/source/CapriKit.Tests/CapriKit.Tests.csproj index 99a7409..d345c16 100644 --- a/source/CapriKit.Tests/CapriKit.Tests.csproj +++ b/source/CapriKit.Tests/CapriKit.Tests.csproj @@ -26,7 +26,4 @@ - - - diff --git a/source/CapriKit.Tests/DirectX11/Buffers/GenericBufferTests.cs b/source/CapriKit.Tests/DirectX11/Buffers/GenericBufferTests.cs index 4c3729d..ec6f046 100644 --- a/source/CapriKit.Tests/DirectX11/Buffers/GenericBufferTests.cs +++ b/source/CapriKit.Tests/DirectX11/Buffers/GenericBufferTests.cs @@ -11,7 +11,7 @@ internal class GenericBufferTests { /// /// Tests StructuredBuffer, RWStructuredBuffer and StagingBuffer through a round-trip of data. - /// + /// [Test] public async Task Mix_Upload_Modify_Download_Staging() { From 1ccf9cf0a217ecdb5d11383e185bda78a1b2aecc Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Sat, 18 Jul 2026 22:44:31 +0200 Subject: [PATCH 08/53] Assets --- .../Shaders/ShaderTranscoder.cs | 2 - .../Shaders/VertexShaderTranscoder.cs | 7 +- source/CapriKit.AssetPipeline/AssetDecoder.cs | 41 +++---- source/CapriKit.AssetPipeline/AssetEncoder.cs | 32 +++--- source/CapriKit.AssetPipeline/AssetManager.cs | 52 ++++++--- .../CapriKit.AssetPipeline/AssetUtilities.cs | 10 +- .../IAssetTranscoder.cs | 37 ++++++ source/CapriKit.IO/IOUtilities.cs | 43 ++++++- .../AssetPipeline/AssetDecoderTests.cs | 4 +- .../AssetPipeline/AssetEncoderTests.cs | 4 +- .../AssetPipeline/DummyTranscoder.cs | 7 +- source/CapriKit.Tests/IO/IOUtilitiesTests.cs | 106 ++++++++++++++++++ 12 files changed, 268 insertions(+), 77 deletions(-) create mode 100644 source/CapriKit.AssetPipeline/IAssetTranscoder.cs create mode 100644 source/CapriKit.Tests/IO/IOUtilitiesTests.cs diff --git a/source/CapriKit.AssetPipeline.DirectX11/Shaders/ShaderTranscoder.cs b/source/CapriKit.AssetPipeline.DirectX11/Shaders/ShaderTranscoder.cs index 2891395..2ba8d01 100644 --- a/source/CapriKit.AssetPipeline.DirectX11/Shaders/ShaderTranscoder.cs +++ b/source/CapriKit.AssetPipeline.DirectX11/Shaders/ShaderTranscoder.cs @@ -7,8 +7,6 @@ namespace CapriKit.AssetPipeline.DirectX11.Shaders; internal static class ShaderTranscoder { - public static IReadOnlySet SupportedExtensions { get; } = new HashSet([".hlsl"]).ToFrozenSet(); - public static void WriteCommon(ShaderByteCode shader, IBufferWriter writer) { writer.Write(shader.EntryPoint); diff --git a/source/CapriKit.AssetPipeline.DirectX11/Shaders/VertexShaderTranscoder.cs b/source/CapriKit.AssetPipeline.DirectX11/Shaders/VertexShaderTranscoder.cs index 1aef183..47e9ec4 100644 --- a/source/CapriKit.AssetPipeline.DirectX11/Shaders/VertexShaderTranscoder.cs +++ b/source/CapriKit.AssetPipeline.DirectX11/Shaders/VertexShaderTranscoder.cs @@ -5,13 +5,12 @@ namespace CapriKit.AssetPipeline.DirectX11.Shaders; -public sealed class VertexShaderTranscoder(Device device) : IAssetEncoder, IAssetDecoder +public sealed class VertexShaderTranscoder(Device device) : IAssetTranscoder> { - public IReadOnlySet SupportedExtensions => ShaderTranscoder.SupportedExtensions; public Guid Id => Guid.Parse("{CA3CB37D-9880-4B61-AB09-EBC17E7533E6}"); public int Version => 1; - public async Task Encode(AssetId id, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer) + public async Task Encode(AssetId id, NoSettings _, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer) { var source = await fileSystem.ReadAllText(id.Path); var includePath = id.Path.Directory; @@ -19,7 +18,7 @@ public async Task Encode(AssetId id, IReadOnlyVirtualFileSystem fileSystem, IBuf ShaderTranscoder.WriteCommon(bytes.Common, writer); } - public IVertexShader Decode(AssetId id, ref SequenceReader reader) + public IVertexShader Decode(AssetId id, NoSettings _, ref SequenceReader reader) { var common = ShaderTranscoder.ReadCommon(ref reader); return ShaderCompiler.CreateVertexShader(new VertexShaderByteCode(common), device); diff --git a/source/CapriKit.AssetPipeline/AssetDecoder.cs b/source/CapriKit.AssetPipeline/AssetDecoder.cs index 952fed5..994d9fc 100644 --- a/source/CapriKit.AssetPipeline/AssetDecoder.cs +++ b/source/CapriKit.AssetPipeline/AssetDecoder.cs @@ -6,39 +6,30 @@ namespace CapriKit.AssetPipeline; -public interface IAssetDecoder -{ - Guid Id { get; } - int Version { get; } - -} - -public interface IAssetDecoder : IAssetDecoder -{ - // Synchronous by design: the envelope owns all file IO and hands the decoder an - // in-memory payload. The reader's buffer is only valid for the duration of the call, - // decoders must copy out anything they want to keep. - TAsset Decode(AssetId id, ref SequenceReader reader); - void HotSwap(TAsset instance, TAsset replacement); -} - internal static class AssetDecoder { - public static async Task> Decode(AssetId id, IVirtualFileSystem fileSystem, IAssetDecoder decoder) + public static async Task> Decode(AssetId id, TSettings settings, + IAssetTranscoder decoder, IVirtualFileSystem fileSystem) + where TSettings : IAssetSettings { - var inputPath = ToEncodedFilePath(id.Path); + var inputPath = ToEncodedFilePath(id); ThrowOnFileNotFound(inputPath, fileSystem); using var input = fileSystem.OpenRead(inputPath); - var payloadLength = await ReadHeader(input, decoder, inputPath); - var asset = await ReadPayload(input, payloadLength, decoder, id); + var payloadLength = await ReadHeader(input, settings, decoder, inputPath); + + // TODO: read settings so we can do a byte-for-byte check they have stayed the same + + var asset = await ReadPayload(input, payloadLength, id, settings, decoder); var dependencies = await ReadDependencies(input); - return new Asset(asset, dependencies); + return new Asset(asset, dependencies); } - private static async Task ReadHeader(Stream input, IAssetDecoder decoder, FilePath path) + private static async Task ReadHeader(Stream input, + TSettings settings, IAssetTranscoder decoder, FilePath path) + where TSettings : IAssetSettings { // Encoder id (16 bytes) + encoder version (4 bytes) + payload length (4 bytes) const int HeaderSizeInBytes = 24; @@ -65,14 +56,16 @@ private static async Task ReadHeader(Stream input, IAssetDecoder deco } } - private static async Task ReadPayload(Stream input, int payloadLength, IAssetDecoder decoder, AssetId id) + private static async Task ReadPayload(Stream input, int payloadLength, + AssetId id, TSettings setting, IAssetTranscoder decoder) + where TSettings : IAssetSettings { var buffer = ArrayPool.Shared.Rent(payloadLength); try { await input.ReadExactlyAsync(buffer.AsMemory(0, payloadLength)); var reader = SequenceReaders.Create(buffer, 0, payloadLength); - return decoder.Decode(id, ref reader); + return decoder.Decode(id, setting, ref reader); } finally { diff --git a/source/CapriKit.AssetPipeline/AssetEncoder.cs b/source/CapriKit.AssetPipeline/AssetEncoder.cs index 735ec54..9ea2a7a 100644 --- a/source/CapriKit.AssetPipeline/AssetEncoder.cs +++ b/source/CapriKit.AssetPipeline/AssetEncoder.cs @@ -6,49 +6,45 @@ namespace CapriKit.AssetPipeline; -public interface IAssetEncoder -{ - IReadOnlySet SupportedExtensions { get; } - Guid Id { get; } - int Version { get; } - - Task Encode(AssetId id, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer); -} - internal sealed class AssetEncoder { - public static async Task Encode(AssetId id, IVirtualFileSystem fileSystem, IAssetEncoder encoder) + public static async Task Encode(AssetId id, TSettings settings, IAssetTranscoder encoder, IVirtualFileSystem fileSystem) + where TSettings : IAssetSettings { ThrowOnFileNotFound(id.Path, fileSystem); - var outputPath = ToEncodedFilePath(id.Path); + var outputPath = ToEncodedFilePath(id); using var output = fileSystem.CreateReadWrite(outputPath); var writer = PipeWriter.Create(output); var spy = fileSystem.SpyOn(); WriteHeader(writer, encoder); // payload length is written in WritePayload - await WritePayload(id, encoder, writer, spy); - WriteDependencies(spy, writer); + + // TODO: write settings so we can verify they have stayed the same + + await WritePayload(writer, id, settings, encoder, spy); + WriteDependencies(writer, spy); await writer.FlushAsync(); await writer.CompleteAsync(); } - private static void WriteHeader(PipeWriter writer, IAssetEncoder encoder) + private static void WriteHeader(PipeWriter writer, IAssetTranscoder encoder) { writer.Write(encoder.Id); writer.Write(encoder.Version); } - private static async Task WritePayload(AssetId id, IAssetEncoder encoder, PipeWriter writer, VirtualFileSystemSpy spy) + private static async Task WritePayload(PipeWriter writer, AssetId id, TSettings settings, IAssetTranscoder encoder, VirtualFileSystemSpy spy) + where TSettings : IAssetSettings { var payload = new ArrayBufferWriter(); - await encoder.Encode(id, spy, payload); + await encoder.Encode(id, settings, spy, payload); writer.Write(payload.WrittenCount); writer.Write(payload.WrittenSpan); } - private static void WriteDependencies(VirtualFileSystemSpy spy, PipeWriter writer) + private static void WriteDependencies(PipeWriter writer, VirtualFileSystemSpy spy) { writer.Write(spy.OpenedFiles.Count); foreach (var dependency in spy.OpenedFiles) @@ -56,6 +52,4 @@ private static void WriteDependencies(VirtualFileSystemSpy spy, PipeWriter write writer.Write(dependency); } } - - } diff --git a/source/CapriKit.AssetPipeline/AssetManager.cs b/source/CapriKit.AssetPipeline/AssetManager.cs index 5349312..66ca05d 100644 --- a/source/CapriKit.AssetPipeline/AssetManager.cs +++ b/source/CapriKit.AssetPipeline/AssetManager.cs @@ -1,30 +1,54 @@ +using CapriKit.IO; + namespace CapriKit.AssetPipeline; public class AssetManager { + private readonly IVirtualFileSystem FileSystem; + private readonly Dictionary Transcoders = []; - private readonly Dictionary Encoders = []; - private readonly Dictionary Decoders = []; + public AssetManager(DirectoryPath rootDirectory) + { + FileSystem = new FileSystem().ScopedTo(rootDirectory); + } + + public AssetManager(IVirtualFileSystem fileSystem) + { + FileSystem = fileSystem; + } - public void RegisterTranscoder(IAssetEncoder encoder, IAssetDecoder decoder) + public void RegisterTranscoder(IAssetTranscoder transcoder) + where TSettings : IAssetSettings { - var typeKey = typeof(T); - Encoders[typeKey] = encoder; - Decoders[typeKey] = decoder; + var typeKey = typeof(TAsset); + Transcoders[typeKey] = transcoder; } - public T Decode(AssetId id) + public void Encode(AssetId id, TSettings settings) + where TSettings : IAssetSettings { - var typeKey = typeof(T); - var encoder = Encoders[typeKey]; - var decoder = Decoders[typeKey]; + var typeKey = typeof(TAsset); + var transcoder = Transcoders[typeKey]; - var extension = new string(id.Path.Extension); - if (encoder.SupportedExtensions.Contains(extension)) - { + throw new NotImplementedException(); + } - } + public TAsset Decode(AssetId id, TSettings settings) + where TSettings : IAssetSettings + { + var typeKey = typeof(TAsset); + var transcoder = Transcoders[typeKey]; throw new NotImplementedException(); } + + // TODO: the current typing and generic constraints are neat for writing transcoders but encoding/decoding requires all + // type parameters as type inference doesn't pick up that TAsset can be derived from that kind of IAssetSettings TSettings is. + /* + AssetManager m; + var settings = new NoSettings(); + m.RegisterTranscoder(transcoder); + m.Encode>(id, settings); + m.Decode>(id, settings); + */ } diff --git a/source/CapriKit.AssetPipeline/AssetUtilities.cs b/source/CapriKit.AssetPipeline/AssetUtilities.cs index 5999b10..1060446 100644 --- a/source/CapriKit.AssetPipeline/AssetUtilities.cs +++ b/source/CapriKit.AssetPipeline/AssetUtilities.cs @@ -4,9 +4,15 @@ namespace CapriKit.AssetPipeline; internal static class AssetUtilities { - public static FilePath ToEncodedFilePath(FilePath path) + public static FilePath ToEncodedFilePath(AssetId id) { - return path + ".cka"; + if (string.IsNullOrEmpty(id.Key)) + { + return $"{id.Path}.cka"; + } + + var key = IOUtilities.EscapeFileName(id.Key); + return $"{id.Path}.{key}.cka"; } public static void ThrowOnFileNotFound(FilePath path, IVirtualFileSystem fileSystem) diff --git a/source/CapriKit.AssetPipeline/IAssetTranscoder.cs b/source/CapriKit.AssetPipeline/IAssetTranscoder.cs new file mode 100644 index 0000000..92b4365 --- /dev/null +++ b/source/CapriKit.AssetPipeline/IAssetTranscoder.cs @@ -0,0 +1,37 @@ +using CapriKit.IO; +using System.Buffers; + +namespace CapriKit.AssetPipeline; + +public readonly struct NoSettings : IAssetSettings +{ + public void Write(IBufferWriter writer) + { + // no-op + } +} + +public interface IAssetSettings +{ + void Write(IBufferWriter writer); +} + +public interface IAssetTranscoder +{ + Guid Id { get; } + int Version { get; } +} + +public interface IAssetTranscoder : IAssetTranscoder + where TSettings : IAssetSettings +{ + // Asynchronous since we expect the encoder to read external files + Task Encode(AssetId id, TSettings settings, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer); + + // Synchronous by design: the envelope owns all file IO and hands the decoder an + // in-memory payload. The reader's buffer is only valid for the duration of the call, + // decoders must copy out anything they want to keep. + TAsset Decode(AssetId id, TSettings settings, ref SequenceReader reader); + + void HotSwap(TAsset instance, TAsset replacement); +} diff --git a/source/CapriKit.IO/IOUtilities.cs b/source/CapriKit.IO/IOUtilities.cs index e59d520..3656a1b 100644 --- a/source/CapriKit.IO/IOUtilities.cs +++ b/source/CapriKit.IO/IOUtilities.cs @@ -1,3 +1,6 @@ +using System.Buffers; +using System.Text; + namespace CapriKit.IO; public static class IOUtilities @@ -5,8 +8,8 @@ public static class IOUtilities private const char DirectorySeperator = '/'; private const char AltDirectorySeperator = '\\'; - private static readonly char[] InvalidFileNameChars = Path.GetInvalidFileNameChars(); - private static readonly char[] InvalidPathChars = Path.GetInvalidPathChars(); + private static readonly SearchValues InvalidFileNameChars = SearchValues.Create(Path.GetInvalidFileNameChars()); + private static readonly SearchValues InvalidPathChars = SearchValues.Create(Path.GetInvalidPathChars()); internal static StringComparison GetOSPathComparisonType() { @@ -77,6 +80,7 @@ public static ReadOnlySpan RemoveTrailingDirectorySeparator(ReadOnlySpan /// Determines if a file name is not just whitespace or contains any invalid character. /// + /// The list of invalid character (and thus the exact behavior of this method) is operating system dependent public static bool IsValidFileName(ReadOnlySpan name) { if (name.IsWhiteSpace()) @@ -84,13 +88,14 @@ public static bool IsValidFileName(ReadOnlySpan name) return false; } - return name.IndexOfAny(InvalidFileNameChars) < 0; + return !name.ContainsAny(InvalidFileNameChars); } /// /// Determines if a path to a file has any invalid characters. An empty /// path is invalid as it does not identify a single file. /// + /// The list of invalid character (and thus the exact behavior of this method) is operating system dependent public static bool IsValidFilePath(ReadOnlySpan path) { if (path.IsWhiteSpace()) @@ -108,9 +113,39 @@ public static bool IsValidFilePath(ReadOnlySpan path) /// Determines if a path contains any invalid characters, note that an empty path is valid /// as it could be a relative path to the current directory /// + /// The list of invalid character (and thus the exact behavior of this method) is operating system dependent public static bool IsValidPath(ReadOnlySpan path) { - return path.IndexOfAny(InvalidPathChars) < 0; + return !path.ContainsAny(InvalidPathChars); + } + + /// + /// Escapes a string so that it can be used as a valid file name + /// + /// The list of invalid character (and thus the exact behavior of this method) is operating system dependent + public static ReadOnlySpan EscapeFileName(ReadOnlySpan name) + { + if (IsValidFileName(name)) + { + return name; + } + + var builder = new StringBuilder(); + foreach (var c in name) + { + if (InvalidFileNameChars.Contains(c)) + { + var hexadecimal = ((int)c).ToString("X2"); + builder.Append('%'); + builder.Append(hexadecimal); + } + else + { + builder.Append(c); + } + } + + return builder.ToString(); } /// diff --git a/source/CapriKit.Tests/AssetPipeline/AssetDecoderTests.cs b/source/CapriKit.Tests/AssetPipeline/AssetDecoderTests.cs index dea0922..31acb68 100644 --- a/source/CapriKit.Tests/AssetPipeline/AssetDecoderTests.cs +++ b/source/CapriKit.Tests/AssetPipeline/AssetDecoderTests.cs @@ -14,8 +14,8 @@ public async Task Decode() var transcoder = new DummyTranscoder(); var id = new AssetId("Main", "hello.txt"); - await AssetEncoder.Encode(id, fileSystem, transcoder); - var envelope = await AssetDecoder.Decode(id, fileSystem, transcoder); + await AssetEncoder.Encode(id, default, transcoder, fileSystem); + var envelope = await AssetDecoder.Decode(id, default, transcoder, fileSystem); FilePath expectedDependency = "hello.txt"; await Assert.That(envelope.Value).IsEqualTo("HÉLLO"); diff --git a/source/CapriKit.Tests/AssetPipeline/AssetEncoderTests.cs b/source/CapriKit.Tests/AssetPipeline/AssetEncoderTests.cs index df49367..e16ff65 100644 --- a/source/CapriKit.Tests/AssetPipeline/AssetEncoderTests.cs +++ b/source/CapriKit.Tests/AssetPipeline/AssetEncoderTests.cs @@ -15,8 +15,8 @@ public async Task Encode() var transcoder = new DummyTranscoder(); var id = new AssetId("Main", "hello.txt"); - await AssetEncoder.Encode(id, fileSystem, transcoder); - var bytes = await fileSystem.ReadAllBytes("hello.txt.cka"); + await AssetEncoder.Encode(id, default, transcoder, fileSystem); + var bytes = await fileSystem.ReadAllBytes("hello.txt.Main.cka"); var reader = new SequenceReader(new ReadOnlySequence(bytes)); var encoderId = reader.ReadGuid(); diff --git a/source/CapriKit.Tests/AssetPipeline/DummyTranscoder.cs b/source/CapriKit.Tests/AssetPipeline/DummyTranscoder.cs index c29a769..56fdd2d 100644 --- a/source/CapriKit.Tests/AssetPipeline/DummyTranscoder.cs +++ b/source/CapriKit.Tests/AssetPipeline/DummyTranscoder.cs @@ -8,19 +8,18 @@ namespace CapriKit.Tests.AssetPipeline; /// /// Uppercases a text file /// -internal sealed class DummyTranscoder : IAssetEncoder, IAssetDecoder +internal sealed class DummyTranscoder : IAssetTranscoder> { - public IReadOnlySet SupportedExtensions { get; } = new HashSet([".txt"]); public Guid Id => Guid.Parse("{B87F41E3-6C33-46E4-802A-3E1E82800E7A}"); public int Version => 1; - public async Task Encode(AssetId id, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer) + public async Task Encode(AssetId id, NoSettings settings, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer) { var text = await fileSystem.ReadAllText(id.Path); writer.Write(text.ToUpperInvariant()); } - public string Decode(AssetId id, ref SequenceReader reader) + public string Decode(AssetId id, NoSettings settings, ref SequenceReader reader) { return reader.ReadString(); } diff --git a/source/CapriKit.Tests/IO/IOUtilitiesTests.cs b/source/CapriKit.Tests/IO/IOUtilitiesTests.cs new file mode 100644 index 0000000..7831269 --- /dev/null +++ b/source/CapriKit.Tests/IO/IOUtilitiesTests.cs @@ -0,0 +1,106 @@ +using CapriKit.IO; + +namespace CapriKit.Tests.IO; + +internal class IOUtilitiesTests +{ + [Test] + public async Task NormalizePathSeparators() + { + var normalized = IOUtilities.NormalizePathSeparators(@"C:\Windows\System32"); + await Assert.That(normalized.ToString()).IsEqualTo("C:/Windows/System32"); + } + + [Test] + public async Task NormalizeDotSegments() + { + var normalized = IOUtilities.NormalizeDotSegments(@"C:\Windows\System32\..\Fonts"); + await Assert.That(normalized.ToString()).IsEqualTo(@"C:\Windows\Fonts"); + } + + [Test] + public async Task Normalize() + { + var normalized = IOUtilities.Normalize(@"C:\Windows\System32\..\Fonts"); + await Assert.That(normalized.ToString()).IsEqualTo("C:/Windows/Fonts"); + } + + [Test] + public async Task AddTrailingDirectorySeparator() + { + var path = IOUtilities.AddTrailingDirectorySeparator("C:/Windows"); + await Assert.That(path.ToString()).IsEqualTo("C:/Windows/"); + } + + [Test] + public async Task RemoveTrailingDirectorySeparator() + { + var path = IOUtilities.RemoveTrailingDirectorySeparator("C:/Windows/"); + await Assert.That(path.ToString()).IsEqualTo("C:/Windows"); + } + + [Test] + public async Task IsValidFileName() + { + await Assert.That(IOUtilities.IsValidFileName("config.cfg")).IsTrue(); + } + + [Test] + public async Task IsValidFileName_InvalidCharacter() + { + await Assert.That(IOUtilities.IsValidFileName("config?.cfg")).IsFalse(); + } + + [Test] + public async Task IsValidFilePath() + { + await Assert.That(IOUtilities.IsValidFilePath("C:/Windows/config.cfg")).IsTrue(); + } + + [Test] + public async Task IsValidFilePath_InvalidCharacter() + { + await Assert.That(IOUtilities.IsValidFilePath("C:/Windows/config?.cfg")).IsFalse(); + } + + [Test] + public async Task IsValidPath() + { + await Assert.That(IOUtilities.IsValidPath("C:/Windows")).IsTrue(); + } + + [Test] + public async Task IsValidPath_InvalidCharacter() + { + await Assert.That(IOUtilities.IsValidPath("C:/Win|dows")).IsFalse(); + } + + [Test] + public async Task EscapeFileName() + { + var escaped = IOUtilities.EscapeFileName("shader.hlsl/VertexMain"); + await Assert.That(escaped.ToString()).IsEqualTo("shader.hlsl%2FVertexMain"); + } + + [Test] + public async Task SearchForDirectoryWithMarker() + { + var id = Path.GetRandomFileName(); + var rootPath = Path.Combine(Path.GetTempPath(), $"{nameof(IOUtilitiesTests)}.{id}"); + var root = new DirectoryPath(rootPath); + var nested = new DirectoryPath(Path.Combine(rootPath, "a", "b")); + try + { + Directory.CreateDirectory(nested); + File.WriteAllText(root.Append(new FilePath("marker.txt")), string.Empty); + + var found = IOUtilities.SearchForDirectoryWithMarker(nested, "marker.txt"); + + await Assert.That(found).IsEqualTo(root); + } + finally + { + new DirectoryInfo(root).Delete(true); + } + } +} From 2a1df68f1c264f8df1dad73bb0a953466df6b578 Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Sun, 19 Jul 2026 20:33:34 +0200 Subject: [PATCH 09/53] wip --- Directory.Packages.props | 1 + .../AssetManagerExtensions.cs | 16 ++++++++++++++++ .../Shaders/ShaderTranscoder.cs | 1 - source/CapriKit.AssetPipeline/AssetDecoder.cs | 16 +++++++++++----- source/CapriKit.AssetPipeline/AssetEncoder.cs | 10 ++++++---- source/CapriKit.AssetPipeline/AssetUtilities.cs | 10 ++++++++++ .../CapriKit.AssetPipeline.csproj | 3 +++ 7 files changed, 47 insertions(+), 10 deletions(-) create mode 100644 source/CapriKit.AssetPipeline.DirectX11/AssetManagerExtensions.cs diff --git a/Directory.Packages.props b/Directory.Packages.props index f0c18e1..c4259df 100644 --- a/Directory.Packages.props +++ b/Directory.Packages.props @@ -15,6 +15,7 @@ + diff --git a/source/CapriKit.AssetPipeline.DirectX11/AssetManagerExtensions.cs b/source/CapriKit.AssetPipeline.DirectX11/AssetManagerExtensions.cs new file mode 100644 index 0000000..a2e0e32 --- /dev/null +++ b/source/CapriKit.AssetPipeline.DirectX11/AssetManagerExtensions.cs @@ -0,0 +1,16 @@ +using CapriKit.DirectX11.Resources.Shaders; + +namespace CapriKit.AssetPipeline.DirectX11; + +public static class AssetManagerExtensions +{ + public static void EncodeVertexShader(this AssetManager assetManager, AssetId id) + { + assetManager.Encode>(id, default); + } + + public static IVertexShader DecodeVertexShader(this AssetManager assetManager, AssetId id) + { + return assetManager.Decode>(id, default); + } +} diff --git a/source/CapriKit.AssetPipeline.DirectX11/Shaders/ShaderTranscoder.cs b/source/CapriKit.AssetPipeline.DirectX11/Shaders/ShaderTranscoder.cs index 2ba8d01..f415349 100644 --- a/source/CapriKit.AssetPipeline.DirectX11/Shaders/ShaderTranscoder.cs +++ b/source/CapriKit.AssetPipeline.DirectX11/Shaders/ShaderTranscoder.cs @@ -1,7 +1,6 @@ using CapriKit.DirectX11.Resources.Shaders; using CapriKit.IO.Buffers; using System.Buffers; -using System.Collections.Frozen; namespace CapriKit.AssetPipeline.DirectX11.Shaders; diff --git a/source/CapriKit.AssetPipeline/AssetDecoder.cs b/source/CapriKit.AssetPipeline/AssetDecoder.cs index 994d9fc..1ec3b10 100644 --- a/source/CapriKit.AssetPipeline/AssetDecoder.cs +++ b/source/CapriKit.AssetPipeline/AssetDecoder.cs @@ -1,7 +1,6 @@ using CapriKit.IO; using CapriKit.IO.Buffers; using System.Buffers; - using static CapriKit.AssetPipeline.AssetUtilities; namespace CapriKit.AssetPipeline; @@ -19,8 +18,6 @@ public static async Task> Decode(AssetId id, TS var payloadLength = await ReadHeader(input, settings, decoder, inputPath); - // TODO: read settings so we can do a byte-for-byte check they have stayed the same - var asset = await ReadPayload(input, payloadLength, id, settings, decoder); var dependencies = await ReadDependencies(input); @@ -31,8 +28,8 @@ private static async Task ReadHeader(Stream input, TSettings settings, IAssetTranscoder decoder, FilePath path) where TSettings : IAssetSettings { - // Encoder id (16 bytes) + encoder version (4 bytes) + payload length (4 bytes) - const int HeaderSizeInBytes = 24; + // Encoder id (16 bytes) + encoder version (4 bytes) + settings hash (16 bytes) + payload length (4 bytes) + const int HeaderSizeInBytes = 40; var buffer = ArrayPool.Shared.Rent(HeaderSizeInBytes); try { @@ -40,6 +37,7 @@ private static async Task ReadHeader(Stream input, var reader = SequenceReaders.Create(buffer, 0, HeaderSizeInBytes); var id = reader.ReadGuid(); var version = reader.ReadInt32(); + var hashOfSettingsUsedToEncode = reader.ReadBytes(16); var payloadLength = reader.ReadInt32(); if (id != decoder.Id || version != decoder.Version) @@ -48,6 +46,14 @@ private static async Task ReadHeader(Stream input, $"Cannot decode {path}, it was encoded by {id} v{version} but the decoder is {decoder.Id} v{decoder.Version}"); } + // Validate that the settings used for encoding, match the settings used for decoding + // by doing a byte-for-byte comparison of the hashes of each. + var hashOfSettingsUsedToDecode = HashSettings(settings); + if (!hashOfSettingsUsedToEncode.SequenceEqual(hashOfSettingsUsedToDecode)) + { + throw new InvalidDataException($"Settings used to encode {id} do not match settings used to decode it"); + } + return payloadLength; } finally diff --git a/source/CapriKit.AssetPipeline/AssetEncoder.cs b/source/CapriKit.AssetPipeline/AssetEncoder.cs index 9ea2a7a..139b9d2 100644 --- a/source/CapriKit.AssetPipeline/AssetEncoder.cs +++ b/source/CapriKit.AssetPipeline/AssetEncoder.cs @@ -18,9 +18,7 @@ public static async Task Encode(AssetId id, TSettings setting var writer = PipeWriter.Create(output); var spy = fileSystem.SpyOn(); - WriteHeader(writer, encoder); // payload length is written in WritePayload - - // TODO: write settings so we can verify they have stayed the same + WriteHeader(writer, encoder, settings); // Payload length is written in WritePayload await WritePayload(writer, id, settings, encoder, spy); WriteDependencies(writer, spy); @@ -29,10 +27,14 @@ public static async Task Encode(AssetId id, TSettings setting await writer.CompleteAsync(); } - private static void WriteHeader(PipeWriter writer, IAssetTranscoder encoder) + private static void WriteHeader(PipeWriter writer, IAssetTranscoder encoder, TSettings settings) + where TSettings : IAssetSettings { writer.Write(encoder.Id); writer.Write(encoder.Version); + + var hash = HashSettings(settings); + writer.Write(hash); } private static async Task WritePayload(PipeWriter writer, AssetId id, TSettings settings, IAssetTranscoder encoder, VirtualFileSystemSpy spy) diff --git a/source/CapriKit.AssetPipeline/AssetUtilities.cs b/source/CapriKit.AssetPipeline/AssetUtilities.cs index 1060446..bf91830 100644 --- a/source/CapriKit.AssetPipeline/AssetUtilities.cs +++ b/source/CapriKit.AssetPipeline/AssetUtilities.cs @@ -1,4 +1,6 @@ using CapriKit.IO; +using System.Buffers; +using System.IO.Hashing; namespace CapriKit.AssetPipeline; @@ -22,4 +24,12 @@ public static void ThrowOnFileNotFound(FilePath path, IVirtualFileSystem fileSys throw new FileNotFoundException(null, path); } } + + public static ReadOnlySpan HashSettings(TSettings settings) + where TSettings : IAssetSettings + { + var payload = new ArrayBufferWriter(); + settings.Write(payload); + return XxHash128.Hash(payload.WrittenSpan); + } } diff --git a/source/CapriKit.AssetPipeline/CapriKit.AssetPipeline.csproj b/source/CapriKit.AssetPipeline/CapriKit.AssetPipeline.csproj index 4e4824a..7dccb16 100644 --- a/source/CapriKit.AssetPipeline/CapriKit.AssetPipeline.csproj +++ b/source/CapriKit.AssetPipeline/CapriKit.AssetPipeline.csproj @@ -1,4 +1,7 @@  + + + From a30625aba4ea3610928982cd6aec902bdf3909bb Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Sun, 19 Jul 2026 22:52:27 +0200 Subject: [PATCH 10/53] wip2 --- .../AssetManagerExtensions.cs | 16 ---- .../Shaders/VertexShaderTranscoder.cs | 10 +++ source/CapriKit.AssetPipeline/AssetDecoder.cs | 66 +++++++++++------ source/CapriKit.AssetPipeline/AssetEncoder.cs | 20 +++-- source/CapriKit.AssetPipeline/AssetManager.cs | 74 +++++++++++++------ .../CapriKit.AssetPipeline/AssetUtilities.cs | 10 --- .../CapriKit.AssetPipeline.csproj | 3 - .../IAssetTranscoder.cs | 8 +- .../AssetPipeline/AssetDecoderTests.cs | 2 +- .../AssetPipeline/AssetEncoderTests.cs | 3 + .../AssetPipeline/AssetManagerTests.cs | 38 ++++++++++ .../AssetPipeline/DummyTranscoder.cs | 12 +++ .../AssetPipeline/RepeatTranscoder.cs | 49 ++++++++++++ 13 files changed, 224 insertions(+), 87 deletions(-) delete mode 100644 source/CapriKit.AssetPipeline.DirectX11/AssetManagerExtensions.cs create mode 100644 source/CapriKit.Tests/AssetPipeline/AssetManagerTests.cs create mode 100644 source/CapriKit.Tests/AssetPipeline/RepeatTranscoder.cs diff --git a/source/CapriKit.AssetPipeline.DirectX11/AssetManagerExtensions.cs b/source/CapriKit.AssetPipeline.DirectX11/AssetManagerExtensions.cs deleted file mode 100644 index a2e0e32..0000000 --- a/source/CapriKit.AssetPipeline.DirectX11/AssetManagerExtensions.cs +++ /dev/null @@ -1,16 +0,0 @@ -using CapriKit.DirectX11.Resources.Shaders; - -namespace CapriKit.AssetPipeline.DirectX11; - -public static class AssetManagerExtensions -{ - public static void EncodeVertexShader(this AssetManager assetManager, AssetId id) - { - assetManager.Encode>(id, default); - } - - public static IVertexShader DecodeVertexShader(this AssetManager assetManager, AssetId id) - { - return assetManager.Decode>(id, default); - } -} diff --git a/source/CapriKit.AssetPipeline.DirectX11/Shaders/VertexShaderTranscoder.cs b/source/CapriKit.AssetPipeline.DirectX11/Shaders/VertexShaderTranscoder.cs index 47e9ec4..eb46c18 100644 --- a/source/CapriKit.AssetPipeline.DirectX11/Shaders/VertexShaderTranscoder.cs +++ b/source/CapriKit.AssetPipeline.DirectX11/Shaders/VertexShaderTranscoder.cs @@ -28,4 +28,14 @@ public void HotSwap(IVertexShader instance, IVertexShader replacement) { instance.HotSwap(replacement); } + + public void WriteSettings(NoSettings settings, IBufferWriter writer) + { + // no-op + } + + public NoSettings ReadSettings(ref SequenceReader reader) + { + return default; + } } diff --git a/source/CapriKit.AssetPipeline/AssetDecoder.cs b/source/CapriKit.AssetPipeline/AssetDecoder.cs index 1ec3b10..883bc92 100644 --- a/source/CapriKit.AssetPipeline/AssetDecoder.cs +++ b/source/CapriKit.AssetPipeline/AssetDecoder.cs @@ -1,13 +1,14 @@ using CapriKit.IO; using CapriKit.IO.Buffers; using System.Buffers; +using System.Buffers.Binary; using static CapriKit.AssetPipeline.AssetUtilities; namespace CapriKit.AssetPipeline; internal static class AssetDecoder { - public static async Task> Decode(AssetId id, TSettings settings, + public static async Task> Decode(AssetId id, IAssetTranscoder decoder, IVirtualFileSystem fileSystem) where TSettings : IAssetSettings { @@ -16,20 +17,18 @@ public static async Task> Decode(AssetId id, TS using var input = fileSystem.OpenRead(inputPath); - var payloadLength = await ReadHeader(input, settings, decoder, inputPath); - - var asset = await ReadPayload(input, payloadLength, id, settings, decoder); + await ReadHeader(input, decoder, inputPath); + var settings = await ReadSettings(input, decoder); + var asset = await ReadPayload(input, id, settings, decoder); var dependencies = await ReadDependencies(input); return new Asset(asset, dependencies); } - private static async Task ReadHeader(Stream input, - TSettings settings, IAssetTranscoder decoder, FilePath path) - where TSettings : IAssetSettings + private static async Task ReadHeader(Stream input, IAssetTranscoder decoder, FilePath path) { - // Encoder id (16 bytes) + encoder version (4 bytes) + settings hash (16 bytes) + payload length (4 bytes) - const int HeaderSizeInBytes = 40; + // Encoder id (16 bytes) + encoder version (4 bytes) + const int HeaderSizeInBytes = 20; var buffer = ArrayPool.Shared.Rent(HeaderSizeInBytes); try { @@ -37,24 +36,30 @@ private static async Task ReadHeader(Stream input, var reader = SequenceReaders.Create(buffer, 0, HeaderSizeInBytes); var id = reader.ReadGuid(); var version = reader.ReadInt32(); - var hashOfSettingsUsedToEncode = reader.ReadBytes(16); - var payloadLength = reader.ReadInt32(); if (id != decoder.Id || version != decoder.Version) { throw new InvalidDataException( $"Cannot decode {path}, it was encoded by {id} v{version} but the decoder is {decoder.Id} v{decoder.Version}"); } + } + finally + { + ArrayPool.Shared.Return(buffer); + } + } - // Validate that the settings used for encoding, match the settings used for decoding - // by doing a byte-for-byte comparison of the hashes of each. - var hashOfSettingsUsedToDecode = HashSettings(settings); - if (!hashOfSettingsUsedToEncode.SequenceEqual(hashOfSettingsUsedToDecode)) - { - throw new InvalidDataException($"Settings used to encode {id} do not match settings used to decode it"); - } - - return payloadLength; + private static async Task ReadSettings(Stream input, + IAssetTranscoder decoder) + where TSettings : IAssetSettings + { + var settingsLength = await ReadInt32(input); + var buffer = ArrayPool.Shared.Rent(settingsLength); + try + { + await input.ReadExactlyAsync(buffer.AsMemory(0, settingsLength)); + var reader = SequenceReaders.Create(buffer, 0, settingsLength); + return decoder.ReadSettings(ref reader); } finally { @@ -62,16 +67,17 @@ private static async Task ReadHeader(Stream input, } } - private static async Task ReadPayload(Stream input, int payloadLength, - AssetId id, TSettings setting, IAssetTranscoder decoder) + private static async Task ReadPayload(Stream input, + AssetId id, TSettings settings, IAssetTranscoder decoder) where TSettings : IAssetSettings { + var payloadLength = await ReadInt32(input); var buffer = ArrayPool.Shared.Rent(payloadLength); try { await input.ReadExactlyAsync(buffer.AsMemory(0, payloadLength)); var reader = SequenceReaders.Create(buffer, 0, payloadLength); - return decoder.Decode(id, setting, ref reader); + return decoder.Decode(id, settings, ref reader); } finally { @@ -101,4 +107,18 @@ private static async Task> ReadDependencies(Stream input) ArrayPool.Shared.Return(buffer); } } + + private static async Task ReadInt32(Stream input) + { + var buffer = ArrayPool.Shared.Rent(sizeof(int)); + try + { + await input.ReadExactlyAsync(buffer.AsMemory(0, sizeof(int))); + return BinaryPrimitives.ReadInt32LittleEndian(buffer); + } + finally + { + ArrayPool.Shared.Return(buffer); + } + } } diff --git a/source/CapriKit.AssetPipeline/AssetEncoder.cs b/source/CapriKit.AssetPipeline/AssetEncoder.cs index 139b9d2..1154351 100644 --- a/source/CapriKit.AssetPipeline/AssetEncoder.cs +++ b/source/CapriKit.AssetPipeline/AssetEncoder.cs @@ -6,7 +6,8 @@ namespace CapriKit.AssetPipeline; -internal sealed class AssetEncoder +// File format: [encoder id][encoder version][settings length][settings][payload length][payload][dependency count][dependencies] +internal static class AssetEncoder { public static async Task Encode(AssetId id, TSettings settings, IAssetTranscoder encoder, IVirtualFileSystem fileSystem) where TSettings : IAssetSettings @@ -18,8 +19,8 @@ public static async Task Encode(AssetId id, TSettings setting var writer = PipeWriter.Create(output); var spy = fileSystem.SpyOn(); - WriteHeader(writer, encoder, settings); // Payload length is written in WritePayload - + WriteHeader(writer, encoder); + WriteSettings(writer, encoder, settings); await WritePayload(writer, id, settings, encoder, spy); WriteDependencies(writer, spy); @@ -27,14 +28,19 @@ public static async Task Encode(AssetId id, TSettings setting await writer.CompleteAsync(); } - private static void WriteHeader(PipeWriter writer, IAssetTranscoder encoder, TSettings settings) - where TSettings : IAssetSettings + private static void WriteHeader(PipeWriter writer, IAssetTranscoder encoder) { writer.Write(encoder.Id); writer.Write(encoder.Version); + } - var hash = HashSettings(settings); - writer.Write(hash); + private static void WriteSettings(PipeWriter writer, IAssetTranscoder transcoder, TSettings settings) + where TSettings : IAssetSettings + { + var buffer = new ArrayBufferWriter(); + transcoder.WriteSettings(settings, buffer); + writer.Write(buffer.WrittenCount); + writer.Write(buffer.WrittenSpan); } private static async Task WritePayload(PipeWriter writer, AssetId id, TSettings settings, IAssetTranscoder encoder, VirtualFileSystemSpy spy) diff --git a/source/CapriKit.AssetPipeline/AssetManager.cs b/source/CapriKit.AssetPipeline/AssetManager.cs index 66ca05d..9af87b8 100644 --- a/source/CapriKit.AssetPipeline/AssetManager.cs +++ b/source/CapriKit.AssetPipeline/AssetManager.cs @@ -5,7 +5,9 @@ namespace CapriKit.AssetPipeline; public class AssetManager { private readonly IVirtualFileSystem FileSystem; - private readonly Dictionary Transcoders = []; + + // Values are IRegisteredTranscoder instances, keyed by typeof(TAsset) + private readonly Dictionary Transcoders = []; public AssetManager(DirectoryPath rootDirectory) { @@ -20,35 +22,63 @@ public AssetManager(IVirtualFileSystem fileSystem) public void RegisterTranscoder(IAssetTranscoder transcoder) where TSettings : IAssetSettings { - var typeKey = typeof(TAsset); - Transcoders[typeKey] = transcoder; + Transcoders[typeof(TAsset)] = new RegisteredTranscoder(transcoder); } - public void Encode(AssetId id, TSettings settings) - where TSettings : IAssetSettings + public Task Encode(AssetId id, IAssetSettings settings) + { + return GetTranscoder().Encode(id, settings, FileSystem); + } + + public Task Encode(AssetId id) { - var typeKey = typeof(TAsset); - var transcoder = Transcoders[typeKey]; + return Encode(id, default(NoSettings)); + } - throw new NotImplementedException(); + public async Task Decode(AssetId id) + { + var asset = await GetTranscoder().Decode(id, FileSystem); + return asset.Value; } - public TAsset Decode(AssetId id, TSettings settings) - where TSettings : IAssetSettings + private IRegisteredTranscoder GetTranscoder() { - var typeKey = typeof(TAsset); - var transcoder = Transcoders[typeKey]; + if (!Transcoders.TryGetValue(typeof(TAsset), out var transcoder)) + { + throw new InvalidOperationException($"No transcoder registered for asset type {typeof(TAsset).Name}"); + } - throw new NotImplementedException(); + return (IRegisteredTranscoder)transcoder; } - // TODO: the current typing and generic constraints are neat for writing transcoders but encoding/decoding requires all - // type parameters as type inference doesn't pick up that TAsset can be derived from that kind of IAssetSettings TSettings is. - /* - AssetManager m; - var settings = new NoSettings(); - m.RegisterTranscoder(transcoder); - m.Encode>(id, settings); - m.Decode>(id, settings); - */ + private interface IRegisteredTranscoder + { + Task Encode(AssetId id, IAssetSettings settings, IVirtualFileSystem fileSystem); + Task> Decode(AssetId id, IVirtualFileSystem fileSystem); + } + + // Bridges the public API, which only knows IAssetSettings, to the transcoder's + // concrete TSettings. This lets callers omit TSettings, C# cannot infer it: type inference + // never flows through generic constraints, only through parameter types. + private sealed class RegisteredTranscoder(IAssetTranscoder transcoder) + : IRegisteredTranscoder + where TSettings : IAssetSettings + { + public Task Encode(AssetId id, IAssetSettings settings, IVirtualFileSystem fileSystem) + { + if (settings is not TSettings typedSettings) + { + throw new ArgumentException( + $"Transcoder for {typeof(TAsset).Name} expects settings of type {typeof(TSettings).Name} but got {settings.GetType().Name}", + nameof(settings)); + } + + return AssetEncoder.Encode(id, typedSettings, transcoder, fileSystem); + } + + public Task> Decode(AssetId id, IVirtualFileSystem fileSystem) + { + return AssetDecoder.Decode(id, transcoder, fileSystem); + } + } } diff --git a/source/CapriKit.AssetPipeline/AssetUtilities.cs b/source/CapriKit.AssetPipeline/AssetUtilities.cs index bf91830..1060446 100644 --- a/source/CapriKit.AssetPipeline/AssetUtilities.cs +++ b/source/CapriKit.AssetPipeline/AssetUtilities.cs @@ -1,6 +1,4 @@ using CapriKit.IO; -using System.Buffers; -using System.IO.Hashing; namespace CapriKit.AssetPipeline; @@ -24,12 +22,4 @@ public static void ThrowOnFileNotFound(FilePath path, IVirtualFileSystem fileSys throw new FileNotFoundException(null, path); } } - - public static ReadOnlySpan HashSettings(TSettings settings) - where TSettings : IAssetSettings - { - var payload = new ArrayBufferWriter(); - settings.Write(payload); - return XxHash128.Hash(payload.WrittenSpan); - } } diff --git a/source/CapriKit.AssetPipeline/CapriKit.AssetPipeline.csproj b/source/CapriKit.AssetPipeline/CapriKit.AssetPipeline.csproj index 7dccb16..4e4824a 100644 --- a/source/CapriKit.AssetPipeline/CapriKit.AssetPipeline.csproj +++ b/source/CapriKit.AssetPipeline/CapriKit.AssetPipeline.csproj @@ -1,7 +1,4 @@  - - - diff --git a/source/CapriKit.AssetPipeline/IAssetTranscoder.cs b/source/CapriKit.AssetPipeline/IAssetTranscoder.cs index 92b4365..107a46d 100644 --- a/source/CapriKit.AssetPipeline/IAssetTranscoder.cs +++ b/source/CapriKit.AssetPipeline/IAssetTranscoder.cs @@ -11,10 +11,7 @@ public void Write(IBufferWriter writer) } } -public interface IAssetSettings -{ - void Write(IBufferWriter writer); -} +public interface IAssetSettings { } public interface IAssetTranscoder { @@ -32,6 +29,7 @@ public interface IAssetTranscoder : IAssetTranscoder // in-memory payload. The reader's buffer is only valid for the duration of the call, // decoders must copy out anything they want to keep. TAsset Decode(AssetId id, TSettings settings, ref SequenceReader reader); - + TSettings ReadSettings(ref SequenceReader reader); + void WriteSettings(TSettings settings, IBufferWriter writer); void HotSwap(TAsset instance, TAsset replacement); } diff --git a/source/CapriKit.Tests/AssetPipeline/AssetDecoderTests.cs b/source/CapriKit.Tests/AssetPipeline/AssetDecoderTests.cs index 31acb68..9a2a53a 100644 --- a/source/CapriKit.Tests/AssetPipeline/AssetDecoderTests.cs +++ b/source/CapriKit.Tests/AssetPipeline/AssetDecoderTests.cs @@ -15,7 +15,7 @@ public async Task Decode() var id = new AssetId("Main", "hello.txt"); await AssetEncoder.Encode(id, default, transcoder, fileSystem); - var envelope = await AssetDecoder.Decode(id, default, transcoder, fileSystem); + var envelope = await AssetDecoder.Decode(id, transcoder, fileSystem); FilePath expectedDependency = "hello.txt"; await Assert.That(envelope.Value).IsEqualTo("HÉLLO"); diff --git a/source/CapriKit.Tests/AssetPipeline/AssetEncoderTests.cs b/source/CapriKit.Tests/AssetPipeline/AssetEncoderTests.cs index e16ff65..567af6c 100644 --- a/source/CapriKit.Tests/AssetPipeline/AssetEncoderTests.cs +++ b/source/CapriKit.Tests/AssetPipeline/AssetEncoderTests.cs @@ -21,6 +21,8 @@ public async Task Encode() var reader = new SequenceReader(new ReadOnlySequence(bytes)); var encoderId = reader.ReadGuid(); var encoderVersion = reader.ReadInt32(); + var settingsLength = reader.ReadInt32(); + reader.Advance(settingsLength); var payloadLength = reader.ReadInt32(); reader.Advance(payloadLength); var dependencyCount = reader.ReadInt32(); @@ -29,6 +31,7 @@ public async Task Encode() await Assert.That(encoderId).IsEqualTo(transcoder.Id); await Assert.That(encoderVersion).IsEqualTo(transcoder.Version); + await Assert.That(settingsLength).IsEqualTo(0); await Assert.That(dependencyCount).IsEqualTo(1); await Assert.That(dependency).IsEqualTo("hello.txt"); await Assert.That(end).IsTrue(); diff --git a/source/CapriKit.Tests/AssetPipeline/AssetManagerTests.cs b/source/CapriKit.Tests/AssetPipeline/AssetManagerTests.cs new file mode 100644 index 0000000..e1fb0bc --- /dev/null +++ b/source/CapriKit.Tests/AssetPipeline/AssetManagerTests.cs @@ -0,0 +1,38 @@ +using CapriKit.AssetPipeline; +using CapriKit.IO; + +namespace CapriKit.Tests.AssetPipeline; + +internal class AssetManagerTests +{ + [Test] + public async Task Decode() + { + var fileSystem = new InMemoryFileSystem(); + await fileSystem.WriteAllText("hello.txt", "héllo"); + var manager = new AssetManager(fileSystem); + manager.RegisterTranscoder(new DummyTranscoder()); + var id = new AssetId("Main", "hello.txt"); + + await manager.Encode(id); + var text = await manager.Decode(id); + + await Assert.That(text).IsEqualTo("HÉLLO"); + } + + [Test] + public async Task Decode_SettingsAreReadFromTheEncodedFile() + { + var fileSystem = new InMemoryFileSystem(); + await fileSystem.WriteAllText("hello.txt", "hey"); + var manager = new AssetManager(fileSystem); + manager.RegisterTranscoder(new RepeatTranscoder()); + var id = new AssetId("Main", "hello.txt"); + + // The asset type is inferred from the settings, decoding requires no settings at all + await manager.Encode(id, new RepeatSettings(3)); + var text = await manager.Decode(id); + + await Assert.That(text).IsEqualTo("heyheyhey"); + } +} diff --git a/source/CapriKit.Tests/AssetPipeline/DummyTranscoder.cs b/source/CapriKit.Tests/AssetPipeline/DummyTranscoder.cs index 56fdd2d..af09a2a 100644 --- a/source/CapriKit.Tests/AssetPipeline/DummyTranscoder.cs +++ b/source/CapriKit.Tests/AssetPipeline/DummyTranscoder.cs @@ -19,6 +19,8 @@ public async Task Encode(AssetId id, NoSettings settings, IReadOnlyVirtu writer.Write(text.ToUpperInvariant()); } + + public string Decode(AssetId id, NoSettings settings, ref SequenceReader reader) { return reader.ReadString(); @@ -28,4 +30,14 @@ public void HotSwap(string instance, string replacement) { throw new NotImplementedException(); } + + public NoSettings ReadSettings(ref SequenceReader reader) + { + return default; + } + + public void WriteSettings(NoSettings settings, IBufferWriter writer) + { + + } } diff --git a/source/CapriKit.Tests/AssetPipeline/RepeatTranscoder.cs b/source/CapriKit.Tests/AssetPipeline/RepeatTranscoder.cs new file mode 100644 index 0000000..37c52dc --- /dev/null +++ b/source/CapriKit.Tests/AssetPipeline/RepeatTranscoder.cs @@ -0,0 +1,49 @@ +using CapriKit.AssetPipeline; +using CapriKit.IO; +using CapriKit.IO.Buffers; +using System.Buffers; + +namespace CapriKit.Tests.AssetPipeline; + +internal readonly record struct RepeatSettings(int Count) : IAssetSettings +{ + public void Write(IBufferWriter writer) + { + writer.Write(Count); + } +} + +/// +/// Repeats the text in a text file times +/// +internal sealed class RepeatTranscoder : IAssetTranscoder +{ + public Guid Id => Guid.Parse("{0F1F51E7-2F2B-4E3B-9C93-15BBB61C1AF4}"); + public int Version => 1; + + public async Task Encode(AssetId id, RepeatSettings settings, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer) + { + var text = await fileSystem.ReadAllText(id.Path); + writer.Write(string.Concat(Enumerable.Repeat(text, settings.Count))); + } + + public string Decode(AssetId id, RepeatSettings settings, ref SequenceReader reader) + { + return reader.ReadString(); + } + + public void HotSwap(string instance, string replacement) + { + throw new NotImplementedException(); + } + + public RepeatSettings ReadSettings(ref SequenceReader reader) + { + return new RepeatSettings(reader.ReadInt32()); + } + + public void WriteSettings(RepeatSettings settings, IBufferWriter writer) + { + writer.Write(settings.Count); + } +} From 260b3e1d0ccdd9b2b4fbd068e25b1086a9936be0 Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Sun, 19 Jul 2026 23:41:07 +0200 Subject: [PATCH 11/53] Simplify asset manager generic type erasure --- .../Shaders/VertexShaderTranscoder.cs | 22 ++----- source/CapriKit.AssetPipeline/AssetDecoder.cs | 16 +++-- source/CapriKit.AssetPipeline/AssetEncoder.cs | 10 ++-- source/CapriKit.AssetPipeline/AssetManager.cs | 48 +++------------ .../IAssetTranscoder.cs | 59 +++++++++++++++---- .../NoSettingsTranscoder.cs | 24 ++++++++ .../AssetPipeline/AssetDecoderTests.cs | 2 +- .../AssetPipeline/AssetEncoderTests.cs | 2 +- 8 files changed, 99 insertions(+), 84 deletions(-) create mode 100644 source/CapriKit.AssetPipeline/NoSettingsTranscoder.cs diff --git a/source/CapriKit.AssetPipeline.DirectX11/Shaders/VertexShaderTranscoder.cs b/source/CapriKit.AssetPipeline.DirectX11/Shaders/VertexShaderTranscoder.cs index eb46c18..ba529ec 100644 --- a/source/CapriKit.AssetPipeline.DirectX11/Shaders/VertexShaderTranscoder.cs +++ b/source/CapriKit.AssetPipeline.DirectX11/Shaders/VertexShaderTranscoder.cs @@ -5,12 +5,10 @@ namespace CapriKit.AssetPipeline.DirectX11.Shaders; -public sealed class VertexShaderTranscoder(Device device) : IAssetTranscoder> +public sealed class VertexShaderTranscoder(Device device) + : NoSettingsTranscoder(Guid.Parse("{CA3CB37D-9880-4B61-AB09-EBC17E7533E6}"), 1) { - public Guid Id => Guid.Parse("{CA3CB37D-9880-4B61-AB09-EBC17E7533E6}"); - public int Version => 1; - - public async Task Encode(AssetId id, NoSettings _, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer) + public async override Task Encode(AssetId id, NoSettings _, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer) { var source = await fileSystem.ReadAllText(id.Path); var includePath = id.Path.Directory; @@ -18,24 +16,14 @@ public async Task Encode(AssetId id, NoSettings _, IReadOnlyVirtu ShaderTranscoder.WriteCommon(bytes.Common, writer); } - public IVertexShader Decode(AssetId id, NoSettings _, ref SequenceReader reader) + public override IVertexShader Decode(AssetId id, NoSettings _, ref SequenceReader reader) { var common = ShaderTranscoder.ReadCommon(ref reader); return ShaderCompiler.CreateVertexShader(new VertexShaderByteCode(common), device); } - public void HotSwap(IVertexShader instance, IVertexShader replacement) + public override void HotSwap(IVertexShader instance, IVertexShader replacement) { instance.HotSwap(replacement); } - - public void WriteSettings(NoSettings settings, IBufferWriter writer) - { - // no-op - } - - public NoSettings ReadSettings(ref SequenceReader reader) - { - return default; - } } diff --git a/source/CapriKit.AssetPipeline/AssetDecoder.cs b/source/CapriKit.AssetPipeline/AssetDecoder.cs index 883bc92..2237619 100644 --- a/source/CapriKit.AssetPipeline/AssetDecoder.cs +++ b/source/CapriKit.AssetPipeline/AssetDecoder.cs @@ -8,9 +8,8 @@ namespace CapriKit.AssetPipeline; internal static class AssetDecoder { - public static async Task> Decode(AssetId id, - IAssetTranscoder decoder, IVirtualFileSystem fileSystem) - where TSettings : IAssetSettings + public static async Task> Decode(AssetId id, + IAssetTranscoder decoder, IVirtualFileSystem fileSystem) { var inputPath = ToEncodedFilePath(id); ThrowOnFileNotFound(inputPath, fileSystem); @@ -49,9 +48,8 @@ private static async Task ReadHeader(Stream input, IAssetTranscoder decoder, Fil } } - private static async Task ReadSettings(Stream input, - IAssetTranscoder decoder) - where TSettings : IAssetSettings + private static async Task> ReadSettings(Stream input, + IAssetTranscoder decoder) { var settingsLength = await ReadInt32(input); var buffer = ArrayPool.Shared.Rent(settingsLength); @@ -67,9 +65,8 @@ private static async Task ReadSettings(Stream inpu } } - private static async Task ReadPayload(Stream input, - AssetId id, TSettings settings, IAssetTranscoder decoder) - where TSettings : IAssetSettings + private static async Task ReadPayload(Stream input, + AssetId id, IAssetSettings settings, IAssetTranscoder decoder) { var payloadLength = await ReadInt32(input); var buffer = ArrayPool.Shared.Rent(payloadLength); @@ -108,6 +105,7 @@ private static async Task> ReadDependencies(Stream input) } } + // TODO: move to an extension method in CapriKit.IO private static async Task ReadInt32(Stream input) { var buffer = ArrayPool.Shared.Rent(sizeof(int)); diff --git a/source/CapriKit.AssetPipeline/AssetEncoder.cs b/source/CapriKit.AssetPipeline/AssetEncoder.cs index 1154351..9d8f58d 100644 --- a/source/CapriKit.AssetPipeline/AssetEncoder.cs +++ b/source/CapriKit.AssetPipeline/AssetEncoder.cs @@ -9,8 +9,7 @@ namespace CapriKit.AssetPipeline; // File format: [encoder id][encoder version][settings length][settings][payload length][payload][dependency count][dependencies] internal static class AssetEncoder { - public static async Task Encode(AssetId id, TSettings settings, IAssetTranscoder encoder, IVirtualFileSystem fileSystem) - where TSettings : IAssetSettings + public static async Task Encode(AssetId id, IAssetSettings settings, IAssetTranscoder encoder, IVirtualFileSystem fileSystem) { ThrowOnFileNotFound(id.Path, fileSystem); var outputPath = ToEncodedFilePath(id); @@ -34,8 +33,8 @@ private static void WriteHeader(PipeWriter writer, IAssetTranscoder encoder) writer.Write(encoder.Version); } - private static void WriteSettings(PipeWriter writer, IAssetTranscoder transcoder, TSettings settings) - where TSettings : IAssetSettings + // Settings are stored in full so that decoding does not require the caller to supply them again + private static void WriteSettings(PipeWriter writer, IAssetTranscoder transcoder, IAssetSettings settings) { var buffer = new ArrayBufferWriter(); transcoder.WriteSettings(settings, buffer); @@ -43,8 +42,7 @@ private static void WriteSettings(PipeWriter writer, IAssetTr writer.Write(buffer.WrittenSpan); } - private static async Task WritePayload(PipeWriter writer, AssetId id, TSettings settings, IAssetTranscoder encoder, VirtualFileSystemSpy spy) - where TSettings : IAssetSettings + private static async Task WritePayload(PipeWriter writer, AssetId id, IAssetSettings settings, IAssetTranscoder encoder, VirtualFileSystemSpy spy) { var payload = new ArrayBufferWriter(); await encoder.Encode(id, settings, spy, payload); diff --git a/source/CapriKit.AssetPipeline/AssetManager.cs b/source/CapriKit.AssetPipeline/AssetManager.cs index 9af87b8..22d498a 100644 --- a/source/CapriKit.AssetPipeline/AssetManager.cs +++ b/source/CapriKit.AssetPipeline/AssetManager.cs @@ -6,8 +6,8 @@ public class AssetManager { private readonly IVirtualFileSystem FileSystem; - // Values are IRegisteredTranscoder instances, keyed by typeof(TAsset) - private readonly Dictionary Transcoders = []; + // Values are IAssetTranscoder instances, keyed by typeof(TAsset) + private readonly Dictionary Transcoders = []; public AssetManager(DirectoryPath rootDirectory) { @@ -19,15 +19,14 @@ public AssetManager(IVirtualFileSystem fileSystem) FileSystem = fileSystem; } - public void RegisterTranscoder(IAssetTranscoder transcoder) - where TSettings : IAssetSettings + public void RegisterTranscoder(IAssetTranscoder transcoder) { - Transcoders[typeof(TAsset)] = new RegisteredTranscoder(transcoder); + Transcoders[typeof(TAsset)] = transcoder; } public Task Encode(AssetId id, IAssetSettings settings) { - return GetTranscoder().Encode(id, settings, FileSystem); + return AssetEncoder.Encode(id, settings, GetTranscoder(), FileSystem); } public Task Encode(AssetId id) @@ -37,48 +36,17 @@ public Task Encode(AssetId id) public async Task Decode(AssetId id) { - var asset = await GetTranscoder().Decode(id, FileSystem); + var asset = await AssetDecoder.Decode(id, GetTranscoder(), FileSystem); return asset.Value; } - private IRegisteredTranscoder GetTranscoder() + private IAssetTranscoder GetTranscoder() { if (!Transcoders.TryGetValue(typeof(TAsset), out var transcoder)) { throw new InvalidOperationException($"No transcoder registered for asset type {typeof(TAsset).Name}"); } - return (IRegisteredTranscoder)transcoder; - } - - private interface IRegisteredTranscoder - { - Task Encode(AssetId id, IAssetSettings settings, IVirtualFileSystem fileSystem); - Task> Decode(AssetId id, IVirtualFileSystem fileSystem); - } - - // Bridges the public API, which only knows IAssetSettings, to the transcoder's - // concrete TSettings. This lets callers omit TSettings, C# cannot infer it: type inference - // never flows through generic constraints, only through parameter types. - private sealed class RegisteredTranscoder(IAssetTranscoder transcoder) - : IRegisteredTranscoder - where TSettings : IAssetSettings - { - public Task Encode(AssetId id, IAssetSettings settings, IVirtualFileSystem fileSystem) - { - if (settings is not TSettings typedSettings) - { - throw new ArgumentException( - $"Transcoder for {typeof(TAsset).Name} expects settings of type {typeof(TSettings).Name} but got {settings.GetType().Name}", - nameof(settings)); - } - - return AssetEncoder.Encode(id, typedSettings, transcoder, fileSystem); - } - - public Task> Decode(AssetId id, IVirtualFileSystem fileSystem) - { - return AssetDecoder.Decode(id, transcoder, fileSystem); - } + return (IAssetTranscoder)transcoder; } } diff --git a/source/CapriKit.AssetPipeline/IAssetTranscoder.cs b/source/CapriKit.AssetPipeline/IAssetTranscoder.cs index 107a46d..dffd516 100644 --- a/source/CapriKit.AssetPipeline/IAssetTranscoder.cs +++ b/source/CapriKit.AssetPipeline/IAssetTranscoder.cs @@ -3,13 +3,7 @@ namespace CapriKit.AssetPipeline; -public readonly struct NoSettings : IAssetSettings -{ - public void Write(IBufferWriter writer) - { - // no-op - } -} +public readonly struct NoSettings : IAssetSettings { } public interface IAssetSettings { } @@ -19,7 +13,23 @@ public interface IAssetTranscoder int Version { get; } } -public interface IAssetTranscoder : IAssetTranscoder +// The pipeline consumes transcoders through this settings-erased view so that AssetManager, +// AssetEncoder and AssetDecoder never need a TSettings type parameter. Callers could not have +// supplied it: type inference never flows through generic constraints, only through parameter +// types. The erased members are internal because only the pipeline should call them; transcoder +// authors implement IAssetTranscoder, which bridges them. +public interface IAssetTranscoder : IAssetTranscoder +{ + internal Task Encode(AssetId id, IAssetSettings settings, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer); + internal TAsset Decode(AssetId id, IAssetSettings settings, ref SequenceReader reader); + internal IAssetSettings ReadSettings(ref SequenceReader reader); + internal void WriteSettings(IAssetSettings settings, IBufferWriter writer); + + // Public and without a bridge, implementations must provide it themselves + void HotSwap(TAsset instance, TAsset replacement); +} + +public interface IAssetTranscoder : IAssetTranscoder where TSettings : IAssetSettings { // Asynchronous since we expect the encoder to read external files @@ -29,7 +39,36 @@ public interface IAssetTranscoder : IAssetTranscoder // in-memory payload. The reader's buffer is only valid for the duration of the call, // decoders must copy out anything they want to keep. TAsset Decode(AssetId id, TSettings settings, ref SequenceReader reader); - TSettings ReadSettings(ref SequenceReader reader); + + // `new` because it hides the erased ReadSettings: same parameters, more specific return type + new TSettings ReadSettings(ref SequenceReader reader); + void WriteSettings(TSettings settings, IBufferWriter writer); - void HotSwap(TAsset instance, TAsset replacement); + + // Default implementations that bridge the settings-erased members to their typed + // counterparts. This is the only place where the pipeline transitions from + // IAssetSettings back to the concrete TSettings + Task IAssetTranscoder.Encode(AssetId id, IAssetSettings settings, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer) + => Encode(id, AsTypedSettings(settings), fileSystem, writer); + + TAsset IAssetTranscoder.Decode(AssetId id, IAssetSettings settings, ref SequenceReader reader) + => Decode(id, AsTypedSettings(settings), ref reader); + + IAssetSettings IAssetTranscoder.ReadSettings(ref SequenceReader reader) + => ReadSettings(ref reader); + + void IAssetTranscoder.WriteSettings(IAssetSettings settings, IBufferWriter writer) + => WriteSettings(AsTypedSettings(settings), writer); + + private static TSettings AsTypedSettings(IAssetSettings settings) + { + if (settings is not TSettings typedSettings) + { + throw new ArgumentException( + $"Transcoder for {typeof(TAsset).Name} expects settings of type {typeof(TSettings).Name} but got {settings?.GetType().Name ?? "null"}", + nameof(settings)); + } + + return typedSettings; + } } diff --git a/source/CapriKit.AssetPipeline/NoSettingsTranscoder.cs b/source/CapriKit.AssetPipeline/NoSettingsTranscoder.cs new file mode 100644 index 0000000..7455592 --- /dev/null +++ b/source/CapriKit.AssetPipeline/NoSettingsTranscoder.cs @@ -0,0 +1,24 @@ +using CapriKit.IO; +using System.Buffers; + +namespace CapriKit.AssetPipeline; + +public abstract class NoSettingsTranscoder(Guid id, int version) : IAssetTranscoder> +{ + public Guid Id { get; } = id; + public int Version { get; } = version; + + public abstract TAsset Decode(AssetId id, NoSettings settings, ref SequenceReader reader); + public abstract Task Encode(AssetId id, NoSettings settings, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer); + public abstract void HotSwap(TAsset instance, TAsset replacement); + + public NoSettings ReadSettings(ref SequenceReader reader) + { + return default; + } + + public void WriteSettings(NoSettings settings, IBufferWriter writer) + { + // no-op + } +} diff --git a/source/CapriKit.Tests/AssetPipeline/AssetDecoderTests.cs b/source/CapriKit.Tests/AssetPipeline/AssetDecoderTests.cs index 9a2a53a..636d68a 100644 --- a/source/CapriKit.Tests/AssetPipeline/AssetDecoderTests.cs +++ b/source/CapriKit.Tests/AssetPipeline/AssetDecoderTests.cs @@ -14,7 +14,7 @@ public async Task Decode() var transcoder = new DummyTranscoder(); var id = new AssetId("Main", "hello.txt"); - await AssetEncoder.Encode(id, default, transcoder, fileSystem); + await AssetEncoder.Encode(id, new NoSettings(), transcoder, fileSystem); var envelope = await AssetDecoder.Decode(id, transcoder, fileSystem); FilePath expectedDependency = "hello.txt"; diff --git a/source/CapriKit.Tests/AssetPipeline/AssetEncoderTests.cs b/source/CapriKit.Tests/AssetPipeline/AssetEncoderTests.cs index 567af6c..8abcc30 100644 --- a/source/CapriKit.Tests/AssetPipeline/AssetEncoderTests.cs +++ b/source/CapriKit.Tests/AssetPipeline/AssetEncoderTests.cs @@ -15,7 +15,7 @@ public async Task Encode() var transcoder = new DummyTranscoder(); var id = new AssetId("Main", "hello.txt"); - await AssetEncoder.Encode(id, default, transcoder, fileSystem); + await AssetEncoder.Encode(id, new NoSettings(), transcoder, fileSystem); var bytes = await fileSystem.ReadAllBytes("hello.txt.Main.cka"); var reader = new SequenceReader(new ReadOnlySequence(bytes)); From 659efa2ccc518a3bcfc31f40c492a13a56955a93 Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Mon, 20 Jul 2026 16:43:04 +0200 Subject: [PATCH 12/53] Clean-up asset manager, add more dependency information --- .../Shaders/ShaderTranscoder.cs | 2 +- source/CapriKit.AssetPipeline/Asset.cs | 2 +- source/CapriKit.AssetPipeline/AssetDecoder.cs | 119 ++++++------------ source/CapriKit.AssetPipeline/AssetEncoder.cs | 9 +- source/CapriKit.AssetPipeline/Dependency.cs | 5 + .../IAssetTranscoder.cs | 8 +- .../IBufferWriterExtensions.cs | 9 +- .../SequenceReaderExtensions.cs | 25 +++- .../AssetPipeline/AssetEncoderTests.cs | 2 +- .../AssetPipeline/DummyTranscoder.cs | 2 +- .../AssetPipeline/RepeatTranscoder.cs | 2 +- .../IBufferWriterExtensionsTests.cs | 4 +- .../SequenceReaderExtensionsTests.cs | 24 +++- 13 files changed, 115 insertions(+), 98 deletions(-) create mode 100644 source/CapriKit.AssetPipeline/Dependency.cs rename source/CapriKit.IO/{Buffers => Streams}/IBufferWriterExtensions.cs (86%) rename source/CapriKit.IO/{Buffers => Streams}/SequenceReaderExtensions.cs (77%) rename source/CapriKit.Tests/IO/{Buffers => Streams}/IBufferWriterExtensionsTests.cs (94%) rename source/CapriKit.Tests/IO/{Buffers => Streams}/SequenceReaderExtensionsTests.cs (77%) diff --git a/source/CapriKit.AssetPipeline.DirectX11/Shaders/ShaderTranscoder.cs b/source/CapriKit.AssetPipeline.DirectX11/Shaders/ShaderTranscoder.cs index f415349..9bfc580 100644 --- a/source/CapriKit.AssetPipeline.DirectX11/Shaders/ShaderTranscoder.cs +++ b/source/CapriKit.AssetPipeline.DirectX11/Shaders/ShaderTranscoder.cs @@ -1,5 +1,5 @@ using CapriKit.DirectX11.Resources.Shaders; -using CapriKit.IO.Buffers; +using CapriKit.IO.Streams; using System.Buffers; namespace CapriKit.AssetPipeline.DirectX11.Shaders; diff --git a/source/CapriKit.AssetPipeline/Asset.cs b/source/CapriKit.AssetPipeline/Asset.cs index 5cbcfbb..d732af6 100644 --- a/source/CapriKit.AssetPipeline/Asset.cs +++ b/source/CapriKit.AssetPipeline/Asset.cs @@ -2,4 +2,4 @@ namespace CapriKit.AssetPipeline; -internal sealed record Asset(T Value, IReadOnlySet Dependencies); +internal sealed record Asset(T Value, IReadOnlyList Dependencies); diff --git a/source/CapriKit.AssetPipeline/AssetDecoder.cs b/source/CapriKit.AssetPipeline/AssetDecoder.cs index 2237619..80345a7 100644 --- a/source/CapriKit.AssetPipeline/AssetDecoder.cs +++ b/source/CapriKit.AssetPipeline/AssetDecoder.cs @@ -1,11 +1,13 @@ using CapriKit.IO; -using CapriKit.IO.Buffers; +using CapriKit.IO.Streams; using System.Buffers; -using System.Buffers.Binary; using static CapriKit.AssetPipeline.AssetUtilities; namespace CapriKit.AssetPipeline; +/// +/// Decodes the generic asset envelope and, using a specialized IAssetTranscoder, the asset itself +/// internal static class AssetDecoder { public static async Task> Decode(AssetId id, @@ -15,32 +17,20 @@ public static async Task> Decode(AssetId id, ThrowOnFileNotFound(inputPath, fileSystem); using var input = fileSystem.OpenRead(inputPath); + var length = checked((int)input.Length); - await ReadHeader(input, decoder, inputPath); - var settings = await ReadSettings(input, decoder); - var asset = await ReadPayload(input, id, settings, decoder); - var dependencies = await ReadDependencies(input); - - return new Asset(asset, dependencies); - } - - private static async Task ReadHeader(Stream input, IAssetTranscoder decoder, FilePath path) - { - // Encoder id (16 bytes) + encoder version (4 bytes) - const int HeaderSizeInBytes = 20; - var buffer = ArrayPool.Shared.Rent(HeaderSizeInBytes); + var buffer = ArrayPool.Shared.Rent(length); try { - await input.ReadExactlyAsync(buffer.AsMemory(0, HeaderSizeInBytes)); - var reader = SequenceReaders.Create(buffer, 0, HeaderSizeInBytes); - var id = reader.ReadGuid(); - var version = reader.ReadInt32(); + await input.ReadExactlyAsync(buffer.AsMemory(0, length)); + var reader = SequenceReaders.Create(buffer, 0, length); + + ReadHeader(ref reader, decoder, inputPath); + var settings = ReadSettings(ref reader, decoder); + var asset = ReadPayload(ref reader, id, settings, decoder); + var dependencies = ReadDependencies(ref reader); - if (id != decoder.Id || version != decoder.Version) - { - throw new InvalidDataException( - $"Cannot decode {path}, it was encoded by {id} v{version} but the decoder is {decoder.Id} v{decoder.Version}"); - } + return new Asset(asset, dependencies); } finally { @@ -48,75 +38,44 @@ private static async Task ReadHeader(Stream input, IAssetTranscoder decoder, Fil } } - private static async Task> ReadSettings(Stream input, - IAssetTranscoder decoder) + private static void ReadHeader(ref SequenceReader reader, IAssetTranscoder decoder, FilePath path) { - var settingsLength = await ReadInt32(input); - var buffer = ArrayPool.Shared.Rent(settingsLength); - try - { - await input.ReadExactlyAsync(buffer.AsMemory(0, settingsLength)); - var reader = SequenceReaders.Create(buffer, 0, settingsLength); - return decoder.ReadSettings(ref reader); - } - finally + var id = reader.ReadGuid(); + var version = reader.ReadInt32(); + + if (id != decoder.Id || version != decoder.Version) { - ArrayPool.Shared.Return(buffer); + throw new InvalidDataException( + $"Cannot decode {path}, it was encoded by {id} v{version} but the decoder is {decoder.Id} v{decoder.Version}"); } } - private static async Task ReadPayload(Stream input, - AssetId id, IAssetSettings settings, IAssetTranscoder decoder) + private static IAssetSettings ReadSettings(ref SequenceReader reader, + IAssetTranscoder decoder) { - var payloadLength = await ReadInt32(input); - var buffer = ArrayPool.Shared.Rent(payloadLength); - try - { - await input.ReadExactlyAsync(buffer.AsMemory(0, payloadLength)); - var reader = SequenceReaders.Create(buffer, 0, payloadLength); - return decoder.Decode(id, settings, ref reader); - } - finally - { - ArrayPool.Shared.Return(buffer); - } + var settingsLength = reader.ReadInt32(); + var settingsReader = reader.SliceUnread(settingsLength); + return decoder.ReadSettings(ref settingsReader); } - private static async Task> ReadDependencies(Stream input) + private static TAsset ReadPayload(ref SequenceReader reader, + AssetId id, IAssetSettings settings, IAssetTranscoder decoder) { - var length = (int)(input.Length - input.Position); - var buffer = ArrayPool.Shared.Rent(length); - try - { - await input.ReadExactlyAsync(buffer.AsMemory(0, length)); - var reader = SequenceReaders.Create(buffer, 0, length); - - var count = reader.ReadInt32(); - var dependencies = new HashSet(count); - for (var i = 0; i < count; i++) - { - dependencies.Add(reader.ReadString()); - } - return dependencies; - } - finally - { - ArrayPool.Shared.Return(buffer); - } + var payloadLength = reader.ReadInt32(); + var payloadReader = reader.SliceUnread(payloadLength); + return decoder.Decode(id, settings, ref payloadReader); } - // TODO: move to an extension method in CapriKit.IO - private static async Task ReadInt32(Stream input) + private static List ReadDependencies(ref SequenceReader reader) { - var buffer = ArrayPool.Shared.Rent(sizeof(int)); - try + var count = reader.ReadInt32(); + var dependencies = new List(count); + for (var i = 0; i < count; i++) { - await input.ReadExactlyAsync(buffer.AsMemory(0, sizeof(int))); - return BinaryPrimitives.ReadInt32LittleEndian(buffer); - } - finally - { - ArrayPool.Shared.Return(buffer); + var lastWriteTicks = reader.ReadInt64(); + var filePath = reader.ReadString(); + dependencies.Add(new Dependency(filePath, new DateTime(lastWriteTicks))); } + return dependencies; } } diff --git a/source/CapriKit.AssetPipeline/AssetEncoder.cs b/source/CapriKit.AssetPipeline/AssetEncoder.cs index 9d8f58d..9434779 100644 --- a/source/CapriKit.AssetPipeline/AssetEncoder.cs +++ b/source/CapriKit.AssetPipeline/AssetEncoder.cs @@ -1,5 +1,5 @@ using CapriKit.IO; -using CapriKit.IO.Buffers; +using CapriKit.IO.Streams; using System.Buffers; using System.IO.Pipelines; using static CapriKit.AssetPipeline.AssetUtilities; @@ -7,6 +7,10 @@ namespace CapriKit.AssetPipeline; // File format: [encoder id][encoder version][settings length][settings][payload length][payload][dependency count][dependencies] + +/// +/// Encodes the generic asset envelope and, using a specialized IAssetTranscoder, the asset itself +/// internal static class AssetEncoder { public static async Task Encode(AssetId id, IAssetSettings settings, IAssetTranscoder encoder, IVirtualFileSystem fileSystem) @@ -33,7 +37,6 @@ private static void WriteHeader(PipeWriter writer, IAssetTranscoder encoder) writer.Write(encoder.Version); } - // Settings are stored in full so that decoding does not require the caller to supply them again private static void WriteSettings(PipeWriter writer, IAssetTranscoder transcoder, IAssetSettings settings) { var buffer = new ArrayBufferWriter(); @@ -55,6 +58,8 @@ private static void WriteDependencies(PipeWriter writer, VirtualFileSystemSpy sp writer.Write(spy.OpenedFiles.Count); foreach (var dependency in spy.OpenedFiles) { + var lastWrite = spy.LastWriteTime(dependency); + writer.Write(lastWrite.Ticks); writer.Write(dependency); } } diff --git a/source/CapriKit.AssetPipeline/Dependency.cs b/source/CapriKit.AssetPipeline/Dependency.cs new file mode 100644 index 0000000..693d4e7 --- /dev/null +++ b/source/CapriKit.AssetPipeline/Dependency.cs @@ -0,0 +1,5 @@ +using CapriKit.IO; + +namespace CapriKit.AssetPipeline; + +internal sealed record Dependency(FilePath File, DateTime LastWrite); diff --git a/source/CapriKit.AssetPipeline/IAssetTranscoder.cs b/source/CapriKit.AssetPipeline/IAssetTranscoder.cs index dffd516..d70b825 100644 --- a/source/CapriKit.AssetPipeline/IAssetTranscoder.cs +++ b/source/CapriKit.AssetPipeline/IAssetTranscoder.cs @@ -14,10 +14,9 @@ public interface IAssetTranscoder } // The pipeline consumes transcoders through this settings-erased view so that AssetManager, -// AssetEncoder and AssetDecoder never need a TSettings type parameter. Callers could not have -// supplied it: type inference never flows through generic constraints, only through parameter -// types. The erased members are internal because only the pipeline should call them; transcoder -// authors implement IAssetTranscoder, which bridges them. +// AssetEncoder and AssetDecoder never need a TSettings type parameter. The erased members are +// internal because only the pipeline should call them; transcoder authors implement +// IAssetTranscoder, which bridges them. public interface IAssetTranscoder : IAssetTranscoder { internal Task Encode(AssetId id, IAssetSettings settings, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer); @@ -25,7 +24,6 @@ public interface IAssetTranscoder : IAssetTranscoder internal IAssetSettings ReadSettings(ref SequenceReader reader); internal void WriteSettings(IAssetSettings settings, IBufferWriter writer); - // Public and without a bridge, implementations must provide it themselves void HotSwap(TAsset instance, TAsset replacement); } diff --git a/source/CapriKit.IO/Buffers/IBufferWriterExtensions.cs b/source/CapriKit.IO/Streams/IBufferWriterExtensions.cs similarity index 86% rename from source/CapriKit.IO/Buffers/IBufferWriterExtensions.cs rename to source/CapriKit.IO/Streams/IBufferWriterExtensions.cs index f84b3f6..673b7a2 100644 --- a/source/CapriKit.IO/Buffers/IBufferWriterExtensions.cs +++ b/source/CapriKit.IO/Streams/IBufferWriterExtensions.cs @@ -3,7 +3,7 @@ using System.Runtime.CompilerServices; using System.Text; -namespace CapriKit.IO.Buffers; +namespace CapriKit.IO.Streams; public static class BufferWriterExtensions { @@ -51,6 +51,13 @@ public static void Write(this IBufferWriter writer, int value) writer.Advance(sizeof(int)); } + public static void Write(this IBufferWriter writer, long value) + { + var span = writer.GetSpan(sizeof(int)); + BinaryPrimitives.WriteInt64LittleEndian(span, value); + writer.Advance(sizeof(int)); + } + public static void Write(this IBufferWriter writer, Guid guid) { var span = writer.GetSpan(Unsafe.SizeOf()); diff --git a/source/CapriKit.IO/Buffers/SequenceReaderExtensions.cs b/source/CapriKit.IO/Streams/SequenceReaderExtensions.cs similarity index 77% rename from source/CapriKit.IO/Buffers/SequenceReaderExtensions.cs rename to source/CapriKit.IO/Streams/SequenceReaderExtensions.cs index 3ea73b9..c25b77e 100644 --- a/source/CapriKit.IO/Buffers/SequenceReaderExtensions.cs +++ b/source/CapriKit.IO/Streams/SequenceReaderExtensions.cs @@ -2,7 +2,7 @@ using System.Runtime.CompilerServices; using System.Text; -namespace CapriKit.IO.Buffers; +namespace CapriKit.IO.Streams; public static class SequenceReaders { @@ -15,7 +15,20 @@ public static SequenceReader Create(byte[] bytes, int start, int length) public static class SequenceReaderExtensions { + /// + /// Creates a new reader to read a slice of the unread sequence. Advances the original reader. + /// Use this method when you want to delegate reading a part of the sequence to another method + /// without giving it access to the entire sequence. + /// + public static SequenceReader SliceUnread(ref this SequenceReader reader, int length) + { + if (!reader.TryReadExact(length, out var slice)) + { + throw new EndOfStreamException(); + } + return new SequenceReader(slice); + } /// /// Reads a length prefixed string written by @@ -78,6 +91,16 @@ public static int ReadInt32(this ref SequenceReader reader) return value; } + public static long ReadInt64(this ref SequenceReader reader) + { + if (!reader.TryReadLittleEndian(out long value)) + { + throw new EndOfStreamException(); + } + + return value; + } + public static Guid ReadGuid(this ref SequenceReader reader) { Span bytes = stackalloc byte[Unsafe.SizeOf()]; diff --git a/source/CapriKit.Tests/AssetPipeline/AssetEncoderTests.cs b/source/CapriKit.Tests/AssetPipeline/AssetEncoderTests.cs index 8abcc30..f1a8b44 100644 --- a/source/CapriKit.Tests/AssetPipeline/AssetEncoderTests.cs +++ b/source/CapriKit.Tests/AssetPipeline/AssetEncoderTests.cs @@ -1,6 +1,6 @@ using CapriKit.AssetPipeline; using CapriKit.IO; -using CapriKit.IO.Buffers; +using CapriKit.IO.Streams; using System.Buffers; namespace CapriKit.Tests.AssetPipeline; diff --git a/source/CapriKit.Tests/AssetPipeline/DummyTranscoder.cs b/source/CapriKit.Tests/AssetPipeline/DummyTranscoder.cs index af09a2a..5619eba 100644 --- a/source/CapriKit.Tests/AssetPipeline/DummyTranscoder.cs +++ b/source/CapriKit.Tests/AssetPipeline/DummyTranscoder.cs @@ -1,6 +1,6 @@ using CapriKit.AssetPipeline; using CapriKit.IO; -using CapriKit.IO.Buffers; +using CapriKit.IO.Streams; using System.Buffers; namespace CapriKit.Tests.AssetPipeline; diff --git a/source/CapriKit.Tests/AssetPipeline/RepeatTranscoder.cs b/source/CapriKit.Tests/AssetPipeline/RepeatTranscoder.cs index 37c52dc..549c482 100644 --- a/source/CapriKit.Tests/AssetPipeline/RepeatTranscoder.cs +++ b/source/CapriKit.Tests/AssetPipeline/RepeatTranscoder.cs @@ -1,6 +1,6 @@ using CapriKit.AssetPipeline; using CapriKit.IO; -using CapriKit.IO.Buffers; +using CapriKit.IO.Streams; using System.Buffers; namespace CapriKit.Tests.AssetPipeline; diff --git a/source/CapriKit.Tests/IO/Buffers/IBufferWriterExtensionsTests.cs b/source/CapriKit.Tests/IO/Streams/IBufferWriterExtensionsTests.cs similarity index 94% rename from source/CapriKit.Tests/IO/Buffers/IBufferWriterExtensionsTests.cs rename to source/CapriKit.Tests/IO/Streams/IBufferWriterExtensionsTests.cs index f0bd49d..4a4f06b 100644 --- a/source/CapriKit.Tests/IO/Buffers/IBufferWriterExtensionsTests.cs +++ b/source/CapriKit.Tests/IO/Streams/IBufferWriterExtensionsTests.cs @@ -1,9 +1,9 @@ -using CapriKit.IO.Buffers; +using CapriKit.IO.Streams; using System.Buffers; using System.Buffers.Binary; using System.Text; -namespace CapriKit.Tests.IO.Buffers; +namespace CapriKit.Tests.IO.Streams; internal class IBufferWriterExtensionsTests { diff --git a/source/CapriKit.Tests/IO/Buffers/SequenceReaderExtensionsTests.cs b/source/CapriKit.Tests/IO/Streams/SequenceReaderExtensionsTests.cs similarity index 77% rename from source/CapriKit.Tests/IO/Buffers/SequenceReaderExtensionsTests.cs rename to source/CapriKit.Tests/IO/Streams/SequenceReaderExtensionsTests.cs index bb7ef01..21e3f45 100644 --- a/source/CapriKit.Tests/IO/Buffers/SequenceReaderExtensionsTests.cs +++ b/source/CapriKit.Tests/IO/Streams/SequenceReaderExtensionsTests.cs @@ -1,14 +1,34 @@ -using CapriKit.IO.Buffers; +using CapriKit.IO.Streams; using System.Buffers; using System.Text; -namespace CapriKit.Tests.IO.Buffers; +namespace CapriKit.Tests.IO.Streams; internal class SequenceReaderExtensionsTests { // SequenceReader is a ref struct, so all reading happens before the first // await and only plain locals cross into the assertions + [Test] + public async Task SliceUnread() + { + var writer = new ArrayBufferWriter(); + writer.Write(111); + writer.Write(222); + + var reader = new SequenceReader(new ReadOnlySequence(writer.WrittenMemory)); + var slice = reader.SliceUnread(sizeof(int)); + var sliced = slice.ReadInt32(); + var sliceEnd = slice.End; + var remaining = reader.ReadInt32(); + var end = reader.End; + + await Assert.That(sliced).IsEqualTo(111); + await Assert.That(sliceEnd).IsTrue(); // the slice cannot see beyond its own section + await Assert.That(remaining).IsEqualTo(222); // the original reader skipped the sliced section + await Assert.That(end).IsTrue(); + } + [Test] public async Task ReadInt32() { From 839e0604957da8d551c472dea4277f67bb697129 Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Mon, 20 Jul 2026 19:14:28 +0200 Subject: [PATCH 13/53] Fix sneaky read/write bug --- source/CapriKit.AssetPipeline/AssetDecoder.cs | 6 ++++-- source/CapriKit.IO/Streams/IBufferWriterExtensions.cs | 4 ++-- source/CapriKit.Tests/AssetPipeline/AssetDecoderTests.cs | 5 ++++- source/CapriKit.Tests/AssetPipeline/AssetEncoderTests.cs | 5 +++++ 4 files changed, 15 insertions(+), 5 deletions(-) diff --git a/source/CapriKit.AssetPipeline/AssetDecoder.cs b/source/CapriKit.AssetPipeline/AssetDecoder.cs index 80345a7..1d00dc1 100644 --- a/source/CapriKit.AssetPipeline/AssetDecoder.cs +++ b/source/CapriKit.AssetPipeline/AssetDecoder.cs @@ -73,8 +73,10 @@ private static List ReadDependencies(ref SequenceReader reader for (var i = 0; i < count; i++) { var lastWriteTicks = reader.ReadInt64(); - var filePath = reader.ReadString(); - dependencies.Add(new Dependency(filePath, new DateTime(lastWriteTicks))); + var lastWrite = new DateTime(lastWriteTicks); + var filePathString = reader.ReadString(); + var filePath = new FilePath(filePathString); + dependencies.Add(new Dependency(filePath, lastWrite)); } return dependencies; } diff --git a/source/CapriKit.IO/Streams/IBufferWriterExtensions.cs b/source/CapriKit.IO/Streams/IBufferWriterExtensions.cs index 673b7a2..da83f09 100644 --- a/source/CapriKit.IO/Streams/IBufferWriterExtensions.cs +++ b/source/CapriKit.IO/Streams/IBufferWriterExtensions.cs @@ -53,9 +53,9 @@ public static void Write(this IBufferWriter writer, int value) public static void Write(this IBufferWriter writer, long value) { - var span = writer.GetSpan(sizeof(int)); + var span = writer.GetSpan(sizeof(long)); BinaryPrimitives.WriteInt64LittleEndian(span, value); - writer.Advance(sizeof(int)); + writer.Advance(sizeof(long)); } public static void Write(this IBufferWriter writer, Guid guid) diff --git a/source/CapriKit.Tests/AssetPipeline/AssetDecoderTests.cs b/source/CapriKit.Tests/AssetPipeline/AssetDecoderTests.cs index 636d68a..80a5cf9 100644 --- a/source/CapriKit.Tests/AssetPipeline/AssetDecoderTests.cs +++ b/source/CapriKit.Tests/AssetPipeline/AssetDecoderTests.cs @@ -18,8 +18,11 @@ public async Task Decode() var envelope = await AssetDecoder.Decode(id, transcoder, fileSystem); FilePath expectedDependency = "hello.txt"; + DateTime expectedTimeStamp = DateTime.Now; await Assert.That(envelope.Value).IsEqualTo("HÉLLO"); await Assert.That(envelope.Dependencies.Count).IsEqualTo(1); - await Assert.That(envelope.Dependencies.First()).IsEqualTo(expectedDependency); + await Assert.That(envelope.Dependencies.First().File).IsEqualTo(expectedDependency); + await Assert.That(envelope.Dependencies.First().LastWrite) + .IsBetween(expectedTimeStamp.AddMinutes(-1), expectedTimeStamp.AddMinutes(1)); } } diff --git a/source/CapriKit.Tests/AssetPipeline/AssetEncoderTests.cs b/source/CapriKit.Tests/AssetPipeline/AssetEncoderTests.cs index f1a8b44..8cb0c48 100644 --- a/source/CapriKit.Tests/AssetPipeline/AssetEncoderTests.cs +++ b/source/CapriKit.Tests/AssetPipeline/AssetEncoderTests.cs @@ -26,13 +26,18 @@ public async Task Encode() var payloadLength = reader.ReadInt32(); reader.Advance(payloadLength); var dependencyCount = reader.ReadInt32(); + var lastWrite = reader.ReadInt64(); var dependency = reader.ReadString(); var end = reader.End; + DateTime expectedTimeStamp = DateTime.Now; + await Assert.That(encoderId).IsEqualTo(transcoder.Id); await Assert.That(encoderVersion).IsEqualTo(transcoder.Version); await Assert.That(settingsLength).IsEqualTo(0); await Assert.That(dependencyCount).IsEqualTo(1); + await Assert.That(new DateTime(lastWrite)) + .IsBetween(expectedTimeStamp.AddMinutes(-1), expectedTimeStamp.AddMinutes(1)); await Assert.That(dependency).IsEqualTo("hello.txt"); await Assert.That(end).IsTrue(); } From 0ff484a69ffa376e31e8369ad9beaacb9ae40b79 Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Mon, 20 Jul 2026 20:14:35 +0200 Subject: [PATCH 14/53] Work on cache --- CLAUDE.md | 6 +- source/CapriKit.AssetPipeline/Asset.cs | 2 +- source/CapriKit.AssetPipeline/AssetCache.cs | 86 +++++++++++++++++++ source/CapriKit.AssetPipeline/AssetManager.cs | 48 ++++++++++- .../CapriKit.AssetPipeline/AssetUtilities.cs | 19 ++++ source/CapriKit.AssetPipeline/Dependency.cs | 2 +- .../AssetPipeline/AssetDecoderTests.cs | 2 +- 7 files changed, 155 insertions(+), 10 deletions(-) create mode 100644 source/CapriKit.AssetPipeline/AssetCache.cs diff --git a/CLAUDE.md b/CLAUDE.md index 31fa598..a70cf56 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -13,10 +13,10 @@ When answering a question keep the following rules in mind: - Code examples are helpful, but they do not have to be complete or to compile. Use a comment like `// snip` to elude boilerplate code. ## Code -If you are asked to generate code,keep changes as small as possible. Focus on creating small, maintainable and easy to understand code. Work in small iterative steps and ensure the test in `Caprikit.Tests` cover the happy path of any code generated and follow the test guidelines found in `source\CapriKit.Tests\README.md`. When generating interop code for C or C++ libraries try to create stateless (functional style) code that mimicks the API of the native library but hides native pointers from the public C# API. Double check that any collection or other action by the garbage collector does not invalidate pointers. +If you are asked to generate code,keep changes as small as possible. Focus on creating small, maintainable and easy to understand code. Work in small iterative steps and ensure the test in `Caprikit.Tests` cover the happy path of any code generated and follow the test guidelines found in `source\CapriKit.Tests\README.md`. When generating interop code for C or C++ libraries try to create stateless (functional style) code that mimics the API of the native library but hides native pointers from the public C# API. Double check that any collection or other action by the garbage collector does not invalidate pointers. ## Documentation -When generating documentation focus on the target audiance: The code maintainers and library users. Write XML documentation to educate the library users on how to use a class, method, property or other component correctly and in which situations it should be used or not be used. Sparingly use in-line comments to explain gotcha's, non-obvious behavior or future points of improvemnts to code maintainers. Remember that documentation also needs to be maintained so focus on writing short to-the-point documentation that is easy for me to maintain. +When generating documentation focus on the target audience: The code maintainers and library users. Write XML documentation to educate the library users on how to use a class, method, property or other component correctly and in which situations it should be used or not be used. Sparingly use in-line comments to explain gotcha's, non-obvious behavior or future points of improvements to code maintainers. Remember that documentation also needs to be maintained so focus on writing short to-the-point documentation that is easy for me to maintain. ## Research In some cases I want to follow-up later on an interesting architecture, coding pattern or tooling suggestion. If I ask you to store or save such an idea for later, create a markdown document in the `research` folder where you explain the suggestion and summarize what we were working on when you suggested it. @@ -27,4 +27,4 @@ Assume you can only use Powershell to execute commands. The `dotnet` tool and la The `external` folder in the root of the repository contains git submodules that point to external repositories. You can depend on code in these external repositories but you can never make changes to them. ## Technologies -See `Directory.Packages.props` for an overview of NuGet packages used. \ No newline at end of file +See `Directory.Packages.props` for an overview of NuGet packages used. diff --git a/source/CapriKit.AssetPipeline/Asset.cs b/source/CapriKit.AssetPipeline/Asset.cs index d732af6..aaa3698 100644 --- a/source/CapriKit.AssetPipeline/Asset.cs +++ b/source/CapriKit.AssetPipeline/Asset.cs @@ -2,4 +2,4 @@ namespace CapriKit.AssetPipeline; -internal sealed record Asset(T Value, IReadOnlyList Dependencies); +public sealed record Asset(T Value, IReadOnlyList Dependencies); diff --git a/source/CapriKit.AssetPipeline/AssetCache.cs b/source/CapriKit.AssetPipeline/AssetCache.cs new file mode 100644 index 0000000..93f1cd2 --- /dev/null +++ b/source/CapriKit.AssetPipeline/AssetCache.cs @@ -0,0 +1,86 @@ +using System.Diagnostics.CodeAnalysis; + +namespace CapriKit.AssetPipeline; + +/// +/// Caches assets in stacked scopes that can be discarded in one go. +/// Assets are released when the scope they were added in is popped or this class is disposed +/// +public sealed class AssetCache : IDisposable +{ + private record CacheItem(int Scope, object Item, IDisposable? Disposable); + + private readonly Dictionary Cache; + private int scope; + + public AssetCache() + { + scope = 0; + Cache = []; + } + + /// + /// Opens a new scope. Assets added from now on are discarded by the matching . + /// + public void PushScope() + { + scope++; + } + + /// + /// Adds an asset to the current scope. + /// + /// Adding an item with the same id twice throws + public void Add(AssetId id, Asset asset) + { + if (Cache.ContainsKey(id)) + { + throw new InvalidOperationException($"Cannot add item with id: {id} a second time"); + } + Cache[id] = new CacheItem(scope, asset, asset.Value as IDisposable); + } + + public bool TryGet(AssetId id, [NotNullWhen(true)] out Asset? asset) + { + if (Cache.TryGetValue(id, out var entry)) + { + asset = (Asset)entry.Item; + return true; + } + + asset = default; + return false; + } + + /// + /// Pops the current scope, discarding and disposing every asset added within it. + /// + public void PopScope() + { + if (scope <= 0) + { + throw new InvalidOperationException("No scope to pop"); + } + + foreach (var (key, value) in Cache) + { + if (value.Scope >= scope) + { + value.Disposable?.Dispose(); + Cache.Remove(key); + } + } + + scope--; + } + + public void Dispose() + { + foreach (var (_, value) in Cache) + { + value.Disposable?.Dispose(); + } + + Cache.Clear(); + } +} diff --git a/source/CapriKit.AssetPipeline/AssetManager.cs b/source/CapriKit.AssetPipeline/AssetManager.cs index 22d498a..1a51655 100644 --- a/source/CapriKit.AssetPipeline/AssetManager.cs +++ b/source/CapriKit.AssetPipeline/AssetManager.cs @@ -1,24 +1,28 @@ using CapriKit.IO; +using static CapriKit.AssetPipeline.AssetUtilities; namespace CapriKit.AssetPipeline; -public class AssetManager +public sealed class AssetManager { private readonly IVirtualFileSystem FileSystem; // Values are IAssetTranscoder instances, keyed by typeof(TAsset) private readonly Dictionary Transcoders = []; + private readonly AssetCache Cache; public AssetManager(DirectoryPath rootDirectory) - { - FileSystem = new FileSystem().ScopedTo(rootDirectory); - } + : this(new FileSystem().ScopedTo(rootDirectory)) { } public AssetManager(IVirtualFileSystem fileSystem) { FileSystem = fileSystem; + Cache = new AssetCache(); } + public void PushScope() => Cache.PushScope(); + public void PopScope() => Cache.PopScope(); + public void RegisterTranscoder(IAssetTranscoder transcoder) { Transcoders[typeof(TAsset)] = transcoder; @@ -40,6 +44,42 @@ public async Task Decode(AssetId id) return asset.Value; } + public async Task Load(AssetId id, IAssetSettings settings, bool rebuildOnFailure = true, bool rebuildOnOutOfDate = true) + { + // If a file was already loaded successfully we do not have to do an out-of-date check. + // Keeping live assets up-to-date is handled by the hot-reloading machinery. + if (Cache.TryGet(id, out var entry)) + { + return entry.Value; + } + + try + { + var asset = await AssetDecoder.Decode(id, GetTranscoder(), FileSystem); + if (!IsUpToDate(asset, FileSystem) && rebuildOnOutOfDate) + { + (asset.Value as IDisposable)?.Dispose(); + await Encode(id, settings); + return await Load(id, settings, false, false); + } + + Cache.Add(id, asset); + return asset.Value; + } + catch (Exception) + { + if (rebuildOnFailure) + { + await Encode(id, settings); + return await Load(id, settings, false, false); + } + else + { + throw; + } + } + } + private IAssetTranscoder GetTranscoder() { if (!Transcoders.TryGetValue(typeof(TAsset), out var transcoder)) diff --git a/source/CapriKit.AssetPipeline/AssetUtilities.cs b/source/CapriKit.AssetPipeline/AssetUtilities.cs index 1060446..79facd8 100644 --- a/source/CapriKit.AssetPipeline/AssetUtilities.cs +++ b/source/CapriKit.AssetPipeline/AssetUtilities.cs @@ -22,4 +22,23 @@ public static void ThrowOnFileNotFound(FilePath path, IVirtualFileSystem fileSys throw new FileNotFoundException(null, path); } } + + public static bool IsUpToDate(Asset asset, IReadOnlyVirtualFileSystem fileSystem) + { + foreach (var (file, version) in asset.Dependencies) + { + if (!fileSystem.Exists(file)) + { + return false; + } + + var lastWrite = fileSystem.LastWriteTime(file); + if (version < lastWrite) + { + return false; + } + } + + return true; + } } diff --git a/source/CapriKit.AssetPipeline/Dependency.cs b/source/CapriKit.AssetPipeline/Dependency.cs index 693d4e7..bf04b1d 100644 --- a/source/CapriKit.AssetPipeline/Dependency.cs +++ b/source/CapriKit.AssetPipeline/Dependency.cs @@ -2,4 +2,4 @@ namespace CapriKit.AssetPipeline; -internal sealed record Dependency(FilePath File, DateTime LastWrite); +public sealed record Dependency(FilePath File, DateTime Version); diff --git a/source/CapriKit.Tests/AssetPipeline/AssetDecoderTests.cs b/source/CapriKit.Tests/AssetPipeline/AssetDecoderTests.cs index 80a5cf9..907b100 100644 --- a/source/CapriKit.Tests/AssetPipeline/AssetDecoderTests.cs +++ b/source/CapriKit.Tests/AssetPipeline/AssetDecoderTests.cs @@ -22,7 +22,7 @@ public async Task Decode() await Assert.That(envelope.Value).IsEqualTo("HÉLLO"); await Assert.That(envelope.Dependencies.Count).IsEqualTo(1); await Assert.That(envelope.Dependencies.First().File).IsEqualTo(expectedDependency); - await Assert.That(envelope.Dependencies.First().LastWrite) + await Assert.That(envelope.Dependencies.First().Version) .IsBetween(expectedTimeStamp.AddMinutes(-1), expectedTimeStamp.AddMinutes(1)); } } From 92e1a66ffcc35df4a07851d8c38cb7ed59bc37b3 Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Mon, 20 Jul 2026 21:36:35 +0200 Subject: [PATCH 15/53] Work on hot reloading --- source/CapriKit.AssetPipeline/AssetManager.cs | 55 ++++-- .../CapriKit.AssetPipeline.csproj | 1 + .../IAssetTranscoder.cs | 5 + source/CapriKit.AssetPipeline/Reloadable.cs | 162 ++++++++++++++++++ 4 files changed, 204 insertions(+), 19 deletions(-) create mode 100644 source/CapriKit.AssetPipeline/Reloadable.cs diff --git a/source/CapriKit.AssetPipeline/AssetManager.cs b/source/CapriKit.AssetPipeline/AssetManager.cs index 1a51655..b5dbd91 100644 --- a/source/CapriKit.AssetPipeline/AssetManager.cs +++ b/source/CapriKit.AssetPipeline/AssetManager.cs @@ -44,40 +44,57 @@ public async Task Decode(AssetId id) return asset.Value; } - public async Task Load(AssetId id, IAssetSettings settings, bool rebuildOnFailure = true, bool rebuildOnOutOfDate = true) + /// + /// Loads an asset from the cache, decoding it from disk and building it first if it is missing or + /// out of date. Loaded assets are owned by the current scope, see . + /// + public async Task Load(AssetId id, IAssetSettings settings) { - // If a file was already loaded successfully we do not have to do an out-of-date check. + // If an asset was already loaded successfully we do not have to do an out-of-date check. // Keeping live assets up-to-date is handled by the hot-reloading machinery. if (Cache.TryGet(id, out var entry)) { return entry.Value; } + var asset = await DecodeOrBuild(id, settings); + Cache.Add(id, asset); + return asset.Value; + } + + public Task Load(AssetId id) + { + return Load(id, default(NoSettings)); + } + + private async Task> DecodeOrBuild(AssetId id, IAssetSettings settings) + { + var transcoder = GetTranscoder(); try { - var asset = await AssetDecoder.Decode(id, GetTranscoder(), FileSystem); - if (!IsUpToDate(asset, FileSystem) && rebuildOnOutOfDate) + var asset = await AssetDecoder.Decode(id, transcoder, FileSystem); + if (IsUpToDate(asset, FileSystem)) { - (asset.Value as IDisposable)?.Dispose(); - await Encode(id, settings); - return await Load(id, settings, false, false); + return asset; } - Cache.Add(id, asset); - return asset.Value; + (asset.Value as IDisposable)?.Dispose(); } - catch (Exception) + catch (Exception ex) when (ex is FileNotFoundException or InvalidDataException) { - if (rebuildOnFailure) - { - await Encode(id, settings); - return await Load(id, settings, false, false); - } - else - { - throw; - } + // The asset was never built, or was built by a different transcoder version } + + // Deliberately not guarded: if what we just built still fails to decode that is a bug in the + // transcoder and the exception should reach the caller + await Encode(id, settings); + return await AssetDecoder.Decode(id, transcoder, FileSystem); + } + + internal void HotSwap(TAsset instance, TAsset replacement) + { + var transcoder = GetTranscoder(); + transcoder.HotSwap(instance, replacement); } private IAssetTranscoder GetTranscoder() diff --git a/source/CapriKit.AssetPipeline/CapriKit.AssetPipeline.csproj b/source/CapriKit.AssetPipeline/CapriKit.AssetPipeline.csproj index 4e4824a..f668ec6 100644 --- a/source/CapriKit.AssetPipeline/CapriKit.AssetPipeline.csproj +++ b/source/CapriKit.AssetPipeline/CapriKit.AssetPipeline.csproj @@ -1,5 +1,6 @@  + diff --git a/source/CapriKit.AssetPipeline/IAssetTranscoder.cs b/source/CapriKit.AssetPipeline/IAssetTranscoder.cs index d70b825..9759f35 100644 --- a/source/CapriKit.AssetPipeline/IAssetTranscoder.cs +++ b/source/CapriKit.AssetPipeline/IAssetTranscoder.cs @@ -24,6 +24,11 @@ public interface IAssetTranscoder : IAssetTranscoder internal IAssetSettings ReadSettings(ref SequenceReader reader); internal void WriteSettings(IAssetSettings settings, IBufferWriter writer); + /// + /// Called when the transcoder must replace the old instance with the replacement. + /// The transcoder is responsible for cleaning-up the old instance and passing + /// ownership of the replacement. + /// void HotSwap(TAsset instance, TAsset replacement); } diff --git a/source/CapriKit.AssetPipeline/Reloadable.cs b/source/CapriKit.AssetPipeline/Reloadable.cs new file mode 100644 index 0000000..b6b4327 --- /dev/null +++ b/source/CapriKit.AssetPipeline/Reloadable.cs @@ -0,0 +1,162 @@ +using CapriKit.Concurrency.Async; +using CapriKit.IO; +using System.Collections.Concurrent; + +namespace CapriKit.AssetPipeline; + +// TODO: fix all outstanding todos and simplify + +internal abstract class Reloadable +{ + public abstract AssetId Id { get; } + public abstract bool IsAlive { get; } + public abstract Task Reload(AssetManager manager, HotSwapQueue queue); +} + +internal sealed class Reloadable : Reloadable + where TAsset : class +{ + private readonly WeakReference Instance; + private readonly IAssetSettings Settings; + + public Reloadable(AssetId id, TAsset asset, IAssetSettings settings) + { + Instance = new WeakReference(asset); + Settings = settings; + Id = id; + } + + public override AssetId Id { get; } + public override bool IsAlive => Instance.TryGetTarget(out _); + + public override async Task Reload(AssetManager manager, HotSwapQueue queue) + { + if (!Instance.TryGetTarget(out var cold)) { return; } + + // TODO: technically cold can be alive but disposed here + + await manager.Encode(Id, Settings); + var hot = await manager.Decode(Id); + queue.Enqueue(cold, hot); + } +} + +internal sealed class HotSwapQueue +{ + private interface IHotSwappable + { + public void HotSwap(AssetManager manager); + } + + private sealed class HotSwappable(TAsset cold, TAsset hot) : IHotSwappable + { + public void HotSwap(AssetManager manager) + { + manager.HotSwap(cold, hot); + } + } + + private readonly ConcurrentQueue Queue = []; + + public void Enqueue(TAsset cold, TAsset hot) + { + Queue.Enqueue(new HotSwappable(cold, hot)); + } + + public void HotSwapPending(AssetManager manager) + { + while (Queue.TryDequeue(out var result)) + { + result.HotSwap(manager); + } + } +} + +internal sealed class HotSwapManager : IDisposable +{ + private readonly AssetManager AssetManager; + private readonly IReadOnlyVirtualFileSystem FileSystem; + private readonly FileSystemEventListener Listener; + private readonly Dictionary> Dependents; + private readonly ConcurrentQueue PendingFileChanges; + private readonly HotSwapQueue HotSwapQueue; + + // TODO: add debounce logic by just refusing to update anything for X time after a change + // so a single DateTime updated by ProcessUpdates while draining the queue + + public HotSwapManager(AssetManager assetManager, IReadOnlyVirtualFileSystem fileSystem) + { + Dependents = []; + PendingFileChanges = []; + HotSwapQueue = new HotSwapQueue(); + + AssetManager = assetManager; + FileSystem = fileSystem; + + // TODO: get the right directory to watch + Listener = new FileSystemEventListener(""); + Listener.OnFileChanged += (sender, @event) => + { + var (target, reason) = @event; + PendingFileChanges.Enqueue(target); + }; + } + + public void Track(AssetId id, Asset asset, IAssetSettings settings) + where TAsset : class + { + var reloadable = new Reloadable(id, asset.Value, settings); + foreach (var dependency in asset.Dependencies) + { + // TODO: file system events give the full path, but our dependencies might + // have a relative path. We need to ensure both are the same or they will never match + var file = dependency.File; + if (Dependents.TryGetValue(file, out var list)) + { + list.Add(reloadable); + } + else + { + Dependents[file] = [reloadable]; + } + } + } + + public void ProcessUpdates() + { + // TODO: pending file changes needs to be debounced + // and multiple file changes might link to one asset change + // so we should first translate this to a hashset of AssetIds we want + // to update, then wait for the debounce window to close, then update all + // assets. + while (PendingFileChanges.TryDequeue(out var result)) + { + if (Dependents.TryGetValue(result, out var dependents)) + { + // TODO: It is also possible that re-encoding is still in progress while we get another + // file change so we should not start more work before FireAndForget marks its task as completed + // (see overload). + + // TODO: after updating we need to 'retrack' as the dependencies can have changed + Update(dependents).FireAndForget(ex => { }); + } + } + + HotSwapQueue.HotSwapPending(AssetManager); + + // TODO: remove all no-longer-alive items + } + + private async Task Update(IReadOnlyList targets) + { + foreach (var target in targets) + { + await target.Reload(AssetManager, HotSwapQueue); + } + } + + public void Dispose() + { + Listener.Dispose(); + } +} From 0cb466d99a377b54bcd51a0e3c811a9f0fc64e49 Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Tue, 21 Jul 2026 10:10:26 +0200 Subject: [PATCH 16/53] Clean-up hot reloading --- .../HotReloading/HotSwapManager.cs | 100 +++++++++++ .../HotReloading/HotSwapQueue.cs | 34 ++++ .../HotReloading/Reloadable.cs | 36 ++++ source/CapriKit.AssetPipeline/Reloadable.cs | 162 ------------------ .../Shaders/ShaderIncludeResolver.cs | 2 +- source/CapriKit.IO/FileSystemEventListener.cs | 15 +- source/CapriKit.IO/FileSystemEventQueue.cs | 35 ++++ source/CapriKit.IO/ScopedFileSystem.cs | 52 ++++-- .../IO/FileSystemEventListenerTests.cs | 6 +- 9 files changed, 260 insertions(+), 182 deletions(-) create mode 100644 source/CapriKit.AssetPipeline/HotReloading/HotSwapManager.cs create mode 100644 source/CapriKit.AssetPipeline/HotReloading/HotSwapQueue.cs create mode 100644 source/CapriKit.AssetPipeline/HotReloading/Reloadable.cs delete mode 100644 source/CapriKit.AssetPipeline/Reloadable.cs create mode 100644 source/CapriKit.IO/FileSystemEventQueue.cs diff --git a/source/CapriKit.AssetPipeline/HotReloading/HotSwapManager.cs b/source/CapriKit.AssetPipeline/HotReloading/HotSwapManager.cs new file mode 100644 index 0000000..bc6ef88 --- /dev/null +++ b/source/CapriKit.AssetPipeline/HotReloading/HotSwapManager.cs @@ -0,0 +1,100 @@ +using CapriKit.Concurrency.Async; +using CapriKit.IO; + +namespace CapriKit.AssetPipeline.HotReloading; + +internal sealed class HotSwapManager : IDisposable +{ + private readonly TimeSpan MinWaitTime = TimeSpan.FromSeconds(0.5); + + private readonly AssetManager AssetManager; + private readonly ReadOnlyScopedFileSystem FileSystem; + private readonly Dictionary> Dependents; + private readonly Dictionary Affected; + + private readonly FileSystemEventQueue PendingFileChanges; + private readonly HotSwapQueue PendingHotSwaps; + + private DateTime lastFileChange; + private int detectedFileChanges; + + public HotSwapManager(AssetManager assetManager, ReadOnlyScopedFileSystem fileSystem) + { + AssetManager = assetManager; + FileSystem = fileSystem; + PendingFileChanges = new FileSystemEventQueue(fileSystem.BasePath); + lastFileChange = DateTime.MinValue; + + Dependents = []; + Affected = []; + PendingHotSwaps = new HotSwapQueue(); + } + + public void Track(AssetId id, Asset asset, IAssetSettings settings) + where TAsset : class + { + var reloadable = new Reloadable(id, asset.Value, settings); + foreach (var dependency in asset.Dependencies) + { + // Match the absolute paths generated by file system events + var file = FileSystem.GetAbsolutePath(dependency.File); + if (Dependents.TryGetValue(file, out var list)) + { + list.Add(reloadable); + } + else + { + Dependents[file] = [reloadable]; + } + } + } + + public void ProcessUpdates() + { + var now = DateTime.Now; + if (PendingFileChanges.Count > detectedFileChanges) + { + detectedFileChanges = PendingFileChanges.Count; + lastFileChange = now; + } + + if (lastFileChange < (now - MinWaitTime)) + { + ProcessAffected(); + } + + PendingHotSwaps.HotSwapPending(AssetManager); + } + + private void ProcessAffected() + { + while (PendingFileChanges.TryDequeue(out var @event) && detectedFileChanges > 0) + { + if (Dependents.TryGetValue(@event.File, out var list)) + { + foreach (var asset in list) + { + Affected.Add(asset.Id, asset); + } + } + --detectedFileChanges; + } + + foreach (var (id, reloadable) in Affected) + { + // TODO: It is also possible that re-encoding is still in progress while we get another + // file change so we should not start more work before FireAndForget marks its task as completed + // (see overload). + + // TODO: after updating we need to 'retrack' as the dependencies can have changed + reloadable.Reload(AssetManager, PendingHotSwaps).FireAndForget(ex => { }); + } + + Affected.Clear(); + } + + public void Dispose() + { + PendingFileChanges.Dispose(); + } +} diff --git a/source/CapriKit.AssetPipeline/HotReloading/HotSwapQueue.cs b/source/CapriKit.AssetPipeline/HotReloading/HotSwapQueue.cs new file mode 100644 index 0000000..306b4d8 --- /dev/null +++ b/source/CapriKit.AssetPipeline/HotReloading/HotSwapQueue.cs @@ -0,0 +1,34 @@ +using System.Collections.Concurrent; + +namespace CapriKit.AssetPipeline.HotReloading; + +internal sealed class HotSwapQueue +{ + private interface IHotSwappable + { + public void HotSwap(AssetManager manager); + } + + private sealed class HotSwappable(TAsset cold, TAsset hot) : IHotSwappable + { + public void HotSwap(AssetManager manager) + { + manager.HotSwap(cold, hot); + } + } + + private readonly ConcurrentQueue Queue = []; + + public void Enqueue(TAsset cold, TAsset hot) + { + Queue.Enqueue(new HotSwappable(cold, hot)); + } + + public void HotSwapPending(AssetManager manager) + { + while (Queue.TryDequeue(out var result)) + { + result.HotSwap(manager); + } + } +} diff --git a/source/CapriKit.AssetPipeline/HotReloading/Reloadable.cs b/source/CapriKit.AssetPipeline/HotReloading/Reloadable.cs new file mode 100644 index 0000000..024e689 --- /dev/null +++ b/source/CapriKit.AssetPipeline/HotReloading/Reloadable.cs @@ -0,0 +1,36 @@ +namespace CapriKit.AssetPipeline.HotReloading; + +internal abstract class Reloadable +{ + public abstract AssetId Id { get; } + public abstract bool IsAlive { get; } + public abstract Task Reload(AssetManager manager, HotSwapQueue queue); +} + +internal sealed class Reloadable : Reloadable + where TAsset : class +{ + private readonly WeakReference Instance; + private readonly IAssetSettings Settings; + + public Reloadable(AssetId id, TAsset asset, IAssetSettings settings) + { + Instance = new WeakReference(asset); + Settings = settings; + Id = id; + } + + public override AssetId Id { get; } + public override bool IsAlive => Instance.TryGetTarget(out _); + + public override async Task Reload(AssetManager manager, HotSwapQueue queue) + { + if (!Instance.TryGetTarget(out var cold)) { return; } + + // TODO: technically cold can be alive but disposed here + + await manager.Encode(Id, Settings); + var hot = await manager.Decode(Id); + queue.Enqueue(cold, hot); + } +} diff --git a/source/CapriKit.AssetPipeline/Reloadable.cs b/source/CapriKit.AssetPipeline/Reloadable.cs deleted file mode 100644 index b6b4327..0000000 --- a/source/CapriKit.AssetPipeline/Reloadable.cs +++ /dev/null @@ -1,162 +0,0 @@ -using CapriKit.Concurrency.Async; -using CapriKit.IO; -using System.Collections.Concurrent; - -namespace CapriKit.AssetPipeline; - -// TODO: fix all outstanding todos and simplify - -internal abstract class Reloadable -{ - public abstract AssetId Id { get; } - public abstract bool IsAlive { get; } - public abstract Task Reload(AssetManager manager, HotSwapQueue queue); -} - -internal sealed class Reloadable : Reloadable - where TAsset : class -{ - private readonly WeakReference Instance; - private readonly IAssetSettings Settings; - - public Reloadable(AssetId id, TAsset asset, IAssetSettings settings) - { - Instance = new WeakReference(asset); - Settings = settings; - Id = id; - } - - public override AssetId Id { get; } - public override bool IsAlive => Instance.TryGetTarget(out _); - - public override async Task Reload(AssetManager manager, HotSwapQueue queue) - { - if (!Instance.TryGetTarget(out var cold)) { return; } - - // TODO: technically cold can be alive but disposed here - - await manager.Encode(Id, Settings); - var hot = await manager.Decode(Id); - queue.Enqueue(cold, hot); - } -} - -internal sealed class HotSwapQueue -{ - private interface IHotSwappable - { - public void HotSwap(AssetManager manager); - } - - private sealed class HotSwappable(TAsset cold, TAsset hot) : IHotSwappable - { - public void HotSwap(AssetManager manager) - { - manager.HotSwap(cold, hot); - } - } - - private readonly ConcurrentQueue Queue = []; - - public void Enqueue(TAsset cold, TAsset hot) - { - Queue.Enqueue(new HotSwappable(cold, hot)); - } - - public void HotSwapPending(AssetManager manager) - { - while (Queue.TryDequeue(out var result)) - { - result.HotSwap(manager); - } - } -} - -internal sealed class HotSwapManager : IDisposable -{ - private readonly AssetManager AssetManager; - private readonly IReadOnlyVirtualFileSystem FileSystem; - private readonly FileSystemEventListener Listener; - private readonly Dictionary> Dependents; - private readonly ConcurrentQueue PendingFileChanges; - private readonly HotSwapQueue HotSwapQueue; - - // TODO: add debounce logic by just refusing to update anything for X time after a change - // so a single DateTime updated by ProcessUpdates while draining the queue - - public HotSwapManager(AssetManager assetManager, IReadOnlyVirtualFileSystem fileSystem) - { - Dependents = []; - PendingFileChanges = []; - HotSwapQueue = new HotSwapQueue(); - - AssetManager = assetManager; - FileSystem = fileSystem; - - // TODO: get the right directory to watch - Listener = new FileSystemEventListener(""); - Listener.OnFileChanged += (sender, @event) => - { - var (target, reason) = @event; - PendingFileChanges.Enqueue(target); - }; - } - - public void Track(AssetId id, Asset asset, IAssetSettings settings) - where TAsset : class - { - var reloadable = new Reloadable(id, asset.Value, settings); - foreach (var dependency in asset.Dependencies) - { - // TODO: file system events give the full path, but our dependencies might - // have a relative path. We need to ensure both are the same or they will never match - var file = dependency.File; - if (Dependents.TryGetValue(file, out var list)) - { - list.Add(reloadable); - } - else - { - Dependents[file] = [reloadable]; - } - } - } - - public void ProcessUpdates() - { - // TODO: pending file changes needs to be debounced - // and multiple file changes might link to one asset change - // so we should first translate this to a hashset of AssetIds we want - // to update, then wait for the debounce window to close, then update all - // assets. - while (PendingFileChanges.TryDequeue(out var result)) - { - if (Dependents.TryGetValue(result, out var dependents)) - { - // TODO: It is also possible that re-encoding is still in progress while we get another - // file change so we should not start more work before FireAndForget marks its task as completed - // (see overload). - - // TODO: after updating we need to 'retrack' as the dependencies can have changed - Update(dependents).FireAndForget(ex => { }); - } - } - - HotSwapQueue.HotSwapPending(AssetManager); - - // TODO: remove all no-longer-alive items - } - - private async Task Update(IReadOnlyList targets) - { - foreach (var target in targets) - { - await target.Reload(AssetManager, HotSwapQueue); - } - } - - public void Dispose() - { - Listener.Dispose(); - } -} diff --git a/source/CapriKit.DirectX11/Resources/Shaders/ShaderIncludeResolver.cs b/source/CapriKit.DirectX11/Resources/Shaders/ShaderIncludeResolver.cs index a7e0e0e..c81643c 100644 --- a/source/CapriKit.DirectX11/Resources/Shaders/ShaderIncludeResolver.cs +++ b/source/CapriKit.DirectX11/Resources/Shaders/ShaderIncludeResolver.cs @@ -14,7 +14,7 @@ private sealed class ShaderStream(FilePath source, byte[] buffer) : MemoryStream private readonly ReadOnlyScopedFileSystem FileSystem; - public ShaderIncludeResolver(IReadOnlyVirtualFileSystem fileSystem, string basePath) + public ShaderIncludeResolver(IReadOnlyVirtualFileSystem fileSystem, DirectoryPath basePath) { FileSystem = new ReadOnlyScopedFileSystem(fileSystem, basePath); } diff --git a/source/CapriKit.IO/FileSystemEventListener.cs b/source/CapriKit.IO/FileSystemEventListener.cs index 241928a..df8d681 100644 --- a/source/CapriKit.IO/FileSystemEventListener.cs +++ b/source/CapriKit.IO/FileSystemEventListener.cs @@ -7,8 +7,15 @@ public enum FileSystemChangeKind Deleted, } -public delegate void FileSystemEventHandler(object sender, (FilePath target, FileSystemChangeKind reason) e); +/// The absolute path to the file affected +/// The kind of change the file underwent +public record FileSystemEvent(FilePath File, FileSystemChangeKind Kind); +public delegate void FileSystemEventHandler(object sender, FileSystemEvent e); + +/// +/// Listens for file changes and notifies interested parties via an event +/// public sealed class FileSystemEventListener : IDisposable { private readonly FileSystemWatcher Watcher; @@ -31,9 +38,9 @@ public FileSystemEventListener(DirectoryPath directory, bool includeSubDirectori EnableRaisingEvents = true, }; - Watcher.Created += (s, e) => onFileChanged?.Invoke(s, (FileSystem.GetFilePath(e.FullPath), FileSystemChangeKind.Created)); - Watcher.Changed += (s, e) => onFileChanged?.Invoke(s, (FileSystem.GetFilePath(e.FullPath), FileSystemChangeKind.Changed)); - Watcher.Deleted += (s, e) => onFileChanged?.Invoke(s, (FileSystem.GetFilePath(e.FullPath), FileSystemChangeKind.Deleted)); + Watcher.Created += (s, e) => onFileChanged?.Invoke(s, new FileSystemEvent(FileSystem.GetFilePath(e.FullPath), FileSystemChangeKind.Created)); + Watcher.Changed += (s, e) => onFileChanged?.Invoke(s, new FileSystemEvent(FileSystem.GetFilePath(e.FullPath), FileSystemChangeKind.Changed)); + Watcher.Deleted += (s, e) => onFileChanged?.Invoke(s, new FileSystemEvent(FileSystem.GetFilePath(e.FullPath), FileSystemChangeKind.Deleted)); } public event FileSystemEventHandler? OnFileChanged diff --git a/source/CapriKit.IO/FileSystemEventQueue.cs b/source/CapriKit.IO/FileSystemEventQueue.cs new file mode 100644 index 0000000..2e59fbe --- /dev/null +++ b/source/CapriKit.IO/FileSystemEventQueue.cs @@ -0,0 +1,35 @@ +using System.Collections.Concurrent; +using System.Diagnostics.CodeAnalysis; + +namespace CapriKit.IO; + +/// +/// Listens for file changes and puts events in a queue so an interested party can control when to handle them. +/// +public sealed class FileSystemEventQueue : IDisposable +{ + private readonly ConcurrentQueue Queue; + private readonly FileSystemEventListener Events; + + public FileSystemEventQueue(DirectoryPath directory, bool includeSubDirectories = true) + { + Queue = new ConcurrentQueue(); + Events = new FileSystemEventListener(directory, includeSubDirectories); + Events.OnFileChanged += (s, e) => + { + Queue.Enqueue(e); + }; + } + + public int Count => Queue.Count; + + public bool TryDequeue([NotNullWhen(true)] out FileSystemEvent? @event) + { + return Queue.TryDequeue(out @event); + } + + public void Dispose() + { + Events.Dispose(); + } +} diff --git a/source/CapriKit.IO/ScopedFileSystem.cs b/source/CapriKit.IO/ScopedFileSystem.cs index 0e62093..594c5a4 100644 --- a/source/CapriKit.IO/ScopedFileSystem.cs +++ b/source/CapriKit.IO/ScopedFileSystem.cs @@ -38,36 +38,64 @@ public void Delete(FilePath file) /// that resolve to directories outside of the scope of this file system results /// in a ForbiddenPathException. /// -public class ReadOnlyScopedFileSystem(IReadOnlyVirtualFileSystem source, DirectoryPath basePath) : IReadOnlyVirtualFileSystem +public class ReadOnlyScopedFileSystem : IReadOnlyVirtualFileSystem { + private readonly IReadOnlyVirtualFileSystem Source; + + public ReadOnlyScopedFileSystem(IReadOnlyVirtualFileSystem source, DirectoryPath basePath) + { + Source = source; + BasePath = basePath.ToAbsolute(); + } + + + /// + /// Gets the absolute path of a file contained in this scoped file system. To be used with IO methods that are + /// not aware of the virtual file system. Throws if the file points to outside the scoped file system, + /// + public FilePath GetAbsolutePath(FilePath file) => GetFilePath(file); + + + /// + /// Gets the absolute path of a directory contained in this scoped file system. To be used with IO methods that are + /// not aware of the virtual file system. Throws if the directory points to outside the scoped file system, + /// + public DirectoryPath GetAbsolutePath(DirectoryPath directory) => GetDirectoryPath(directory); + + /// + /// The absolute path of the directory this file system is scoped to. + /// + public DirectoryPath BasePath { get; } + + public bool Exists(FilePath file) { - return source.Exists(GetFilePath(file)); + return Source.Exists(GetFilePath(file)); } public DateTime LastWriteTime(FilePath file) { - return source.LastWriteTime(GetFilePath(file)); + return Source.LastWriteTime(GetFilePath(file)); } public IReadOnlyList List(DirectoryPath directory) { - return source.List(GetDirectoryPath(directory)); + return Source.List(GetDirectoryPath(directory)); } public Stream OpenRead(FilePath file) { - return source.OpenRead(GetFilePath(file)); + return Source.OpenRead(GetFilePath(file)); } public long SizeInBytes(FilePath file) { - return source.SizeInBytes(GetFilePath(file)); + return Source.SizeInBytes(GetFilePath(file)); } protected DirectoryPath GetDirectoryPath(DirectoryPath path) { - var fullPath = path.GetPathRelativeTo(basePath); + var fullPath = path.GetPathRelativeTo(BasePath); ThrowIfPathIsOutsideBasePath(fullPath); return fullPath; @@ -75,7 +103,7 @@ protected DirectoryPath GetDirectoryPath(DirectoryPath path) protected FilePath GetFilePath(FilePath path) { - var fullPath = path.GetPathRelativeTo(basePath); + var fullPath = path.GetPathRelativeTo(BasePath); ThrowIfPathIsOutsideBasePath(fullPath); return fullPath; @@ -84,18 +112,18 @@ protected FilePath GetFilePath(FilePath path) protected void ThrowIfPathIsOutsideBasePath(FilePath file) { Debug.Assert(file.IsAbsolute); - if (!file.StartsWith(basePath)) + if (!file.StartsWith(BasePath)) { - throw new ForbiddenPathException(file, basePath); + throw new ForbiddenPathException(file, BasePath); } } protected void ThrowIfPathIsOutsideBasePath(DirectoryPath path) { Debug.Assert(path.IsAbsolute); - if (!path.StartsWith(basePath)) + if (!path.StartsWith(BasePath)) { - throw new ForbiddenPathException(path, basePath); + throw new ForbiddenPathException(path, BasePath); } } } diff --git a/source/CapriKit.Tests/IO/FileSystemEventListenerTests.cs b/source/CapriKit.Tests/IO/FileSystemEventListenerTests.cs index cbfd980..5cd4859 100644 --- a/source/CapriKit.Tests/IO/FileSystemEventListenerTests.cs +++ b/source/CapriKit.Tests/IO/FileSystemEventListenerTests.cs @@ -35,17 +35,17 @@ public async Task OnFileChanged() using var watcher = new FileSystemEventListener(TempDirectory, false); watcher.OnFileChanged += (s, e) => { - if (e.reason == FileSystemChangeKind.Created && e.target.FileName.Equals(fileName, StringComparison.OrdinalIgnoreCase)) + if (e.Kind == FileSystemChangeKind.Created && e.File.FileName.Equals(fileName, StringComparison.OrdinalIgnoreCase)) { created.TrySetResult(); } - if (e.reason == FileSystemChangeKind.Changed && e.target.FileName.Equals(fileName, StringComparison.OrdinalIgnoreCase)) + if (e.Kind == FileSystemChangeKind.Changed && e.File.FileName.Equals(fileName, StringComparison.OrdinalIgnoreCase)) { changed.TrySetResult(); } - if (e.reason == FileSystemChangeKind.Deleted && e.target.FileName.Equals(fileName, StringComparison.OrdinalIgnoreCase)) + if (e.Kind == FileSystemChangeKind.Deleted && e.File.FileName.Equals(fileName, StringComparison.OrdinalIgnoreCase)) { deleted.TrySetResult(); } From fddfcbfde99b4358c7f48a9e11702cbb9722ebbb Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Tue, 21 Jul 2026 12:09:53 +0200 Subject: [PATCH 17/53] Even more clean-up --- Directory.Packages.props | 1 + source/CapriKit.AssetPipeline/Asset.cs | 2 +- source/CapriKit.AssetPipeline/AssetCache.cs | 9 +- source/CapriKit.AssetPipeline/AssetDecoder.cs | 2 +- source/CapriKit.AssetPipeline/AssetManager.cs | 9 +- .../CapriKit.AssetPipeline.csproj | 3 + .../HotReloading/HotSwapManager.cs | 121 +++++++++++------- .../HotReloading/HotSwapQueue.cs | 34 ----- .../HotReloading/HotSwappable.cs | 24 ++++ .../HotReloading/Reloadable.cs | 33 +++-- .../IAssetTranscoder.cs | 10 +- 11 files changed, 147 insertions(+), 101 deletions(-) delete mode 100644 source/CapriKit.AssetPipeline/HotReloading/HotSwapQueue.cs create mode 100644 source/CapriKit.AssetPipeline/HotReloading/HotSwappable.cs diff --git a/Directory.Packages.props b/Directory.Packages.props index c4259df..e52ace0 100644 --- a/Directory.Packages.props +++ b/Directory.Packages.props @@ -12,6 +12,7 @@ + diff --git a/source/CapriKit.AssetPipeline/Asset.cs b/source/CapriKit.AssetPipeline/Asset.cs index aaa3698..3d0f2c0 100644 --- a/source/CapriKit.AssetPipeline/Asset.cs +++ b/source/CapriKit.AssetPipeline/Asset.cs @@ -2,4 +2,4 @@ namespace CapriKit.AssetPipeline; -public sealed record Asset(T Value, IReadOnlyList Dependencies); +public sealed record Asset(AssetId Id, T Value, IReadOnlyList Dependencies); diff --git a/source/CapriKit.AssetPipeline/AssetCache.cs b/source/CapriKit.AssetPipeline/AssetCache.cs index 93f1cd2..ceaab65 100644 --- a/source/CapriKit.AssetPipeline/AssetCache.cs +++ b/source/CapriKit.AssetPipeline/AssetCache.cs @@ -31,20 +31,21 @@ public void PushScope() /// Adds an asset to the current scope. /// /// Adding an item with the same id twice throws - public void Add(AssetId id, Asset asset) + public void Add(AssetId id, T asset) + where T : class { if (Cache.ContainsKey(id)) { throw new InvalidOperationException($"Cannot add item with id: {id} a second time"); } - Cache[id] = new CacheItem(scope, asset, asset.Value as IDisposable); + Cache[id] = new CacheItem(scope, asset, asset as IDisposable); } - public bool TryGet(AssetId id, [NotNullWhen(true)] out Asset? asset) + public bool TryGet(AssetId id, [NotNullWhen(true)] out T? asset) { if (Cache.TryGetValue(id, out var entry)) { - asset = (Asset)entry.Item; + asset = (T)entry.Item; return true; } diff --git a/source/CapriKit.AssetPipeline/AssetDecoder.cs b/source/CapriKit.AssetPipeline/AssetDecoder.cs index 1d00dc1..5a928f2 100644 --- a/source/CapriKit.AssetPipeline/AssetDecoder.cs +++ b/source/CapriKit.AssetPipeline/AssetDecoder.cs @@ -30,7 +30,7 @@ public static async Task> Decode(AssetId id, var asset = ReadPayload(ref reader, id, settings, decoder); var dependencies = ReadDependencies(ref reader); - return new Asset(asset, dependencies); + return new Asset(id, asset, dependencies); } finally { diff --git a/source/CapriKit.AssetPipeline/AssetManager.cs b/source/CapriKit.AssetPipeline/AssetManager.cs index b5dbd91..7129f0b 100644 --- a/source/CapriKit.AssetPipeline/AssetManager.cs +++ b/source/CapriKit.AssetPipeline/AssetManager.cs @@ -40,10 +40,15 @@ public Task Encode(AssetId id) public async Task Decode(AssetId id) { - var asset = await AssetDecoder.Decode(id, GetTranscoder(), FileSystem); + var asset = await DecodeInternal(id); return asset.Value; } + internal Task> DecodeInternal(AssetId id) + { + return AssetDecoder.Decode(id, GetTranscoder(), FileSystem); + } + /// /// Loads an asset from the cache, decoding it from disk and building it first if it is missing or /// out of date. Loaded assets are owned by the current scope, see . @@ -54,7 +59,7 @@ public async Task Load(AssetId id, IAssetSettings settin // Keeping live assets up-to-date is handled by the hot-reloading machinery. if (Cache.TryGet(id, out var entry)) { - return entry.Value; + return entry; } var asset = await DecodeOrBuild(id, settings); diff --git a/source/CapriKit.AssetPipeline/CapriKit.AssetPipeline.csproj b/source/CapriKit.AssetPipeline/CapriKit.AssetPipeline.csproj index f668ec6..f1cb5b1 100644 --- a/source/CapriKit.AssetPipeline/CapriKit.AssetPipeline.csproj +++ b/source/CapriKit.AssetPipeline/CapriKit.AssetPipeline.csproj @@ -1,4 +1,7 @@  + + + diff --git a/source/CapriKit.AssetPipeline/HotReloading/HotSwapManager.cs b/source/CapriKit.AssetPipeline/HotReloading/HotSwapManager.cs index bc6ef88..3bec31c 100644 --- a/source/CapriKit.AssetPipeline/HotReloading/HotSwapManager.cs +++ b/source/CapriKit.AssetPipeline/HotReloading/HotSwapManager.cs @@ -1,100 +1,129 @@ using CapriKit.Concurrency.Async; using CapriKit.IO; +using Microsoft.Extensions.Logging; +using System.Collections.Concurrent; +using System.Diagnostics; namespace CapriKit.AssetPipeline.HotReloading; -internal sealed class HotSwapManager : IDisposable +/// +/// Facilitates hot reloading and hot swapping of assets. Tracks the files used to create an asset +/// and triggers a rebuild on file changes. Takes care of threading and only performs the final +/// hot swap when is called from the main thread. +/// +internal sealed partial class HotSwapManager : IDisposable { private readonly TimeSpan MinWaitTime = TimeSpan.FromSeconds(0.5); + private readonly ILogger Logger; + private readonly AssetManager AssetManager; private readonly ReadOnlyScopedFileSystem FileSystem; - private readonly Dictionary> Dependents; - private readonly Dictionary Affected; + + private readonly Dictionary Tracked; + private readonly Dictionary> Dependents; private readonly FileSystemEventQueue PendingFileChanges; - private readonly HotSwapQueue PendingHotSwaps; + private readonly HashSet PendingReloads; + private readonly ConcurrentQueue PendingHotSwaps; - private DateTime lastFileChange; - private int detectedFileChanges; + private long lastFileChange; - public HotSwapManager(AssetManager assetManager, ReadOnlyScopedFileSystem fileSystem) + public HotSwapManager(ILogger logger, AssetManager assetManager, ReadOnlyScopedFileSystem fileSystem) { + Logger = logger; AssetManager = assetManager; FileSystem = fileSystem; - PendingFileChanges = new FileSystemEventQueue(fileSystem.BasePath); - lastFileChange = DateTime.MinValue; - + Tracked = []; Dependents = []; - Affected = []; - PendingHotSwaps = new HotSwapQueue(); + + PendingFileChanges = new FileSystemEventQueue(fileSystem.BasePath); + PendingReloads = []; + PendingHotSwaps = []; + lastFileChange = Stopwatch.GetTimestamp(); } - public void Track(AssetId id, Asset asset, IAssetSettings settings) + public void Track(Asset asset, IAssetSettings settings) where TAsset : class { - var reloadable = new Reloadable(id, asset.Value, settings); + Tracked[asset.Id] = new Reloadable(asset.Id, asset.Value, settings); foreach (var dependency in asset.Dependencies) { // Match the absolute paths generated by file system events var file = FileSystem.GetAbsolutePath(dependency.File); - if (Dependents.TryGetValue(file, out var list)) - { - list.Add(reloadable); - } - else - { - Dependents[file] = [reloadable]; - } + if (!Dependents.TryGetValue(file, out var ids)) { ids = Dependents[file] = []; } + ids.Add(asset.Id); } } public void ProcessUpdates() { - var now = DateTime.Now; - if (PendingFileChanges.Count > detectedFileChanges) - { - detectedFileChanges = PendingFileChanges.Count; - lastFileChange = now; - } - - if (lastFileChange < (now - MinWaitTime)) + DrainFileChanges(); + var elapsed = Stopwatch.GetElapsedTime(lastFileChange); + if (elapsed > MinWaitTime) { - ProcessAffected(); + ReloadAffected(); } - PendingHotSwaps.HotSwapPending(AssetManager); + HotSwapCompleted(); } - private void ProcessAffected() + private void DrainFileChanges() { - while (PendingFileChanges.TryDequeue(out var @event) && detectedFileChanges > 0) + while (PendingFileChanges.TryDequeue(out var @event)) { - if (Dependents.TryGetValue(@event.File, out var list)) + if (Dependents.TryGetValue(@event.File, out var dependents)) { - foreach (var asset in list) + lastFileChange = Stopwatch.GetTimestamp(); + foreach (var assetId in dependents) { - Affected.Add(asset.Id, asset); + LogPendingReload(Logger, @event.File, assetId); + PendingReloads.Add(assetId); } } - --detectedFileChanges; } + } - foreach (var (id, reloadable) in Affected) + private void ReloadAffected() + { + PendingReloads.RemoveWhere(id => { - // TODO: It is also possible that re-encoding is still in progress while we get another - // file change so we should not start more work before FireAndForget marks its task as completed - // (see overload). + var reloadable = Tracked[id]; + // Prevent kicking off a reload while the asset is still being reloaded + if (reloadable.IsReloading) { return false; } - // TODO: after updating we need to 'retrack' as the dependencies can have changed - reloadable.Reload(AssetManager, PendingHotSwaps).FireAndForget(ex => { }); - } + LogReloadStarted(Logger, id); + reloadable.Reload(AssetManager, PendingHotSwaps).FireAndForget( + ex => LogReloadFailed(Logger, id, ex), + () => LogReloadCompleted(Logger, id)); - Affected.Clear(); + return true; + }); + } + + private void HotSwapCompleted() + { + while (PendingHotSwaps.TryDequeue(out var result)) + { + result.HotSwap(AssetManager, this); + LogReloadCompleted(Logger, result.Id); + } } public void Dispose() { PendingFileChanges.Dispose(); } + + [LoggerMessage(Level = LogLevel.Information, Message = "Detected file change: {path}, affecting asset: {asset}")] + private static partial void LogPendingReload(ILogger logger, FilePath path, AssetId asset); + + [LoggerMessage(Level = LogLevel.Information, Message = "Reloading asset started: {asset}")] + private static partial void LogReloadStarted(ILogger logger, AssetId asset); + + [LoggerMessage(Level = LogLevel.Information, Message = "Reloading asset completed: {asset}")] + private static partial void LogReloadCompleted(ILogger logger, AssetId asset); + + [LoggerMessage(Level = LogLevel.Error, Message = "Reloading asset failed: {asset}")] + private static partial void LogReloadFailed(ILogger logger, AssetId asset, Exception exception); } diff --git a/source/CapriKit.AssetPipeline/HotReloading/HotSwapQueue.cs b/source/CapriKit.AssetPipeline/HotReloading/HotSwapQueue.cs deleted file mode 100644 index 306b4d8..0000000 --- a/source/CapriKit.AssetPipeline/HotReloading/HotSwapQueue.cs +++ /dev/null @@ -1,34 +0,0 @@ -using System.Collections.Concurrent; - -namespace CapriKit.AssetPipeline.HotReloading; - -internal sealed class HotSwapQueue -{ - private interface IHotSwappable - { - public void HotSwap(AssetManager manager); - } - - private sealed class HotSwappable(TAsset cold, TAsset hot) : IHotSwappable - { - public void HotSwap(AssetManager manager) - { - manager.HotSwap(cold, hot); - } - } - - private readonly ConcurrentQueue Queue = []; - - public void Enqueue(TAsset cold, TAsset hot) - { - Queue.Enqueue(new HotSwappable(cold, hot)); - } - - public void HotSwapPending(AssetManager manager) - { - while (Queue.TryDequeue(out var result)) - { - result.HotSwap(manager); - } - } -} diff --git a/source/CapriKit.AssetPipeline/HotReloading/HotSwappable.cs b/source/CapriKit.AssetPipeline/HotReloading/HotSwappable.cs new file mode 100644 index 0000000..0db06b0 --- /dev/null +++ b/source/CapriKit.AssetPipeline/HotReloading/HotSwappable.cs @@ -0,0 +1,24 @@ +namespace CapriKit.AssetPipeline.HotReloading; + +/// +/// Represents an asset that is ready to be hot swapped by the main thread +/// +internal abstract class HotSwappable(AssetId id) +{ + public AssetId Id { get; } = id; + public abstract void HotSwap(AssetManager manager, HotSwapManager hotSwapManager); +} + +internal sealed class HotSwappable(TAsset instance, Asset newParts, IAssetSettings settings) + : HotSwappable(newParts.Id) + where TAsset : class +{ + public override void HotSwap(AssetManager assetManager, HotSwapManager hotSwapManager) + { + assetManager.HotSwap(instance, newParts.Value); + + // Instance keeps being the active object, so keep tracking instance, but with the new dependencies + var toTrack = new Asset(newParts.Id, instance, newParts.Dependencies); + hotSwapManager.Track(toTrack, settings); + } +} diff --git a/source/CapriKit.AssetPipeline/HotReloading/Reloadable.cs b/source/CapriKit.AssetPipeline/HotReloading/Reloadable.cs index 024e689..199b89e 100644 --- a/source/CapriKit.AssetPipeline/HotReloading/Reloadable.cs +++ b/source/CapriKit.AssetPipeline/HotReloading/Reloadable.cs @@ -1,10 +1,18 @@ +using System.Collections.Concurrent; + namespace CapriKit.AssetPipeline.HotReloading; +/// +/// Represents an asset that can be rebuild and reloaded on demand. +/// internal abstract class Reloadable { + protected volatile bool isReloading; + public abstract AssetId Id { get; } - public abstract bool IsAlive { get; } - public abstract Task Reload(AssetManager manager, HotSwapQueue queue); + public abstract Task Reload(AssetManager manager, ConcurrentQueue queue); + + public bool IsReloading => isReloading; } internal sealed class Reloadable : Reloadable @@ -21,16 +29,23 @@ public Reloadable(AssetId id, TAsset asset, IAssetSettings settings) } public override AssetId Id { get; } - public override bool IsAlive => Instance.TryGetTarget(out _); - public override async Task Reload(AssetManager manager, HotSwapQueue queue) + public override async Task Reload(AssetManager manager, ConcurrentQueue queue) { - if (!Instance.TryGetTarget(out var cold)) { return; } + try + { + isReloading = true; + if (!Instance.TryGetTarget(out var cold)) { return; } - // TODO: technically cold can be alive but disposed here + // TODO: technically cold can be alive but disposed here - await manager.Encode(Id, Settings); - var hot = await manager.Decode(Id); - queue.Enqueue(cold, hot); + await manager.Encode(Id, Settings); + var hot = await manager.DecodeInternal(Id); + queue.Enqueue(new HotSwappable(cold, hot, Settings)); + } + finally + { + isReloading = false; + } } } diff --git a/source/CapriKit.AssetPipeline/IAssetTranscoder.cs b/source/CapriKit.AssetPipeline/IAssetTranscoder.cs index 9759f35..6985e61 100644 --- a/source/CapriKit.AssetPipeline/IAssetTranscoder.cs +++ b/source/CapriKit.AssetPipeline/IAssetTranscoder.cs @@ -25,11 +25,13 @@ public interface IAssetTranscoder : IAssetTranscoder internal void WriteSettings(IAssetSettings settings, IBufferWriter writer); /// - /// Called when the transcoder must replace the old instance with the replacement. - /// The transcoder is responsible for cleaning-up the old instance and passing - /// ownership of the replacement. + /// Moves the contents of into . + /// keeps its identity and stays the live object that callers + /// already hold references to. The transcoder is responsible for cleaning-up any + /// orphaned resources. After calling this method must no longer + /// be used or referenced. /// - void HotSwap(TAsset instance, TAsset replacement); + void HotSwap(TAsset instance, TAsset newParts); } public interface IAssetTranscoder : IAssetTranscoder From e8ff96ccd058f325e75426b7e86571c65398d55e Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Tue, 21 Jul 2026 13:54:55 +0200 Subject: [PATCH 18/53] Fix more content pipeline bugs --- .../Shaders/VertexShaderTranscoder.cs | 4 ++-- source/CapriKit.AssetPipeline/Asset.cs | 5 ++--- source/CapriKit.AssetPipeline/AssetCache.cs | 1 + source/CapriKit.AssetPipeline/AssetDecoder.cs | 1 + source/CapriKit.AssetPipeline/AssetManager.cs | 8 +++++++- source/CapriKit.AssetPipeline/AssetUtilities.cs | 1 + .../CapriKit.AssetPipeline/HotReloading/HotSwapManager.cs | 5 ++--- source/CapriKit.AssetPipeline/NoSettingsTranscoder.cs | 2 +- .../CapriKit.DirectX11/Resources/Shaders/VertexShader.cs | 6 +++--- .../DirectX11/Buffers/GenericBufferTests.cs | 3 ++- 10 files changed, 22 insertions(+), 14 deletions(-) diff --git a/source/CapriKit.AssetPipeline.DirectX11/Shaders/VertexShaderTranscoder.cs b/source/CapriKit.AssetPipeline.DirectX11/Shaders/VertexShaderTranscoder.cs index ba529ec..dc3f23a 100644 --- a/source/CapriKit.AssetPipeline.DirectX11/Shaders/VertexShaderTranscoder.cs +++ b/source/CapriKit.AssetPipeline.DirectX11/Shaders/VertexShaderTranscoder.cs @@ -22,8 +22,8 @@ public override IVertexShader Decode(AssetId id, NoSettings _, re return ShaderCompiler.CreateVertexShader(new VertexShaderByteCode(common), device); } - public override void HotSwap(IVertexShader instance, IVertexShader replacement) + public override void HotSwap(IVertexShader instance, IVertexShader newParts) { - instance.HotSwap(replacement); + instance.HotSwap(newParts); } } diff --git a/source/CapriKit.AssetPipeline/Asset.cs b/source/CapriKit.AssetPipeline/Asset.cs index 3d0f2c0..cc805d8 100644 --- a/source/CapriKit.AssetPipeline/Asset.cs +++ b/source/CapriKit.AssetPipeline/Asset.cs @@ -1,5 +1,4 @@ -using CapriKit.IO; - namespace CapriKit.AssetPipeline; -public sealed record Asset(AssetId Id, T Value, IReadOnlyList Dependencies); +public sealed record Asset(AssetId Id, T Value, IReadOnlyList Dependencies) + where T : class; diff --git a/source/CapriKit.AssetPipeline/AssetCache.cs b/source/CapriKit.AssetPipeline/AssetCache.cs index ceaab65..f6c4202 100644 --- a/source/CapriKit.AssetPipeline/AssetCache.cs +++ b/source/CapriKit.AssetPipeline/AssetCache.cs @@ -42,6 +42,7 @@ public void Add(AssetId id, T asset) } public bool TryGet(AssetId id, [NotNullWhen(true)] out T? asset) + where T : class { if (Cache.TryGetValue(id, out var entry)) { diff --git a/source/CapriKit.AssetPipeline/AssetDecoder.cs b/source/CapriKit.AssetPipeline/AssetDecoder.cs index 5a928f2..710accf 100644 --- a/source/CapriKit.AssetPipeline/AssetDecoder.cs +++ b/source/CapriKit.AssetPipeline/AssetDecoder.cs @@ -12,6 +12,7 @@ internal static class AssetDecoder { public static async Task> Decode(AssetId id, IAssetTranscoder decoder, IVirtualFileSystem fileSystem) + where TAsset : class { var inputPath = ToEncodedFilePath(id); ThrowOnFileNotFound(inputPath, fileSystem); diff --git a/source/CapriKit.AssetPipeline/AssetManager.cs b/source/CapriKit.AssetPipeline/AssetManager.cs index 7129f0b..31b7ff6 100644 --- a/source/CapriKit.AssetPipeline/AssetManager.cs +++ b/source/CapriKit.AssetPipeline/AssetManager.cs @@ -34,17 +34,20 @@ public Task Encode(AssetId id, IAssetSettings settings) } public Task Encode(AssetId id) + where TAsset : class { return Encode(id, default(NoSettings)); } public async Task Decode(AssetId id) + where TAsset : class { var asset = await DecodeInternal(id); return asset.Value; } internal Task> DecodeInternal(AssetId id) + where TAsset : class { return AssetDecoder.Decode(id, GetTranscoder(), FileSystem); } @@ -54,6 +57,7 @@ internal Task> DecodeInternal(AssetId id) /// out of date. Loaded assets are owned by the current scope, see . /// public async Task Load(AssetId id, IAssetSettings settings) + where TAsset : class { // If an asset was already loaded successfully we do not have to do an out-of-date check. // Keeping live assets up-to-date is handled by the hot-reloading machinery. @@ -63,16 +67,18 @@ public async Task Load(AssetId id, IAssetSettings settin } var asset = await DecodeOrBuild(id, settings); - Cache.Add(id, asset); + Cache.Add(id, asset.Value); return asset.Value; } public Task Load(AssetId id) + where TAsset : class { return Load(id, default(NoSettings)); } private async Task> DecodeOrBuild(AssetId id, IAssetSettings settings) + where TAsset : class { var transcoder = GetTranscoder(); try diff --git a/source/CapriKit.AssetPipeline/AssetUtilities.cs b/source/CapriKit.AssetPipeline/AssetUtilities.cs index 79facd8..c0fe1fe 100644 --- a/source/CapriKit.AssetPipeline/AssetUtilities.cs +++ b/source/CapriKit.AssetPipeline/AssetUtilities.cs @@ -24,6 +24,7 @@ public static void ThrowOnFileNotFound(FilePath path, IVirtualFileSystem fileSys } public static bool IsUpToDate(Asset asset, IReadOnlyVirtualFileSystem fileSystem) + where T : class { foreach (var (file, version) in asset.Dependencies) { diff --git a/source/CapriKit.AssetPipeline/HotReloading/HotSwapManager.cs b/source/CapriKit.AssetPipeline/HotReloading/HotSwapManager.cs index 3bec31c..e2bdb6c 100644 --- a/source/CapriKit.AssetPipeline/HotReloading/HotSwapManager.cs +++ b/source/CapriKit.AssetPipeline/HotReloading/HotSwapManager.cs @@ -13,7 +13,7 @@ namespace CapriKit.AssetPipeline.HotReloading; /// internal sealed partial class HotSwapManager : IDisposable { - private readonly TimeSpan MinWaitTime = TimeSpan.FromSeconds(0.5); + private static readonly TimeSpan MinWaitTime = TimeSpan.FromSeconds(0.5); private readonly ILogger Logger; @@ -94,8 +94,7 @@ private void ReloadAffected() LogReloadStarted(Logger, id); reloadable.Reload(AssetManager, PendingHotSwaps).FireAndForget( - ex => LogReloadFailed(Logger, id, ex), - () => LogReloadCompleted(Logger, id)); + ex => LogReloadFailed(Logger, id, ex)); return true; }); diff --git a/source/CapriKit.AssetPipeline/NoSettingsTranscoder.cs b/source/CapriKit.AssetPipeline/NoSettingsTranscoder.cs index 7455592..d0a1e24 100644 --- a/source/CapriKit.AssetPipeline/NoSettingsTranscoder.cs +++ b/source/CapriKit.AssetPipeline/NoSettingsTranscoder.cs @@ -10,7 +10,7 @@ public abstract class NoSettingsTranscoder(Guid id, int version) : IAsse public abstract TAsset Decode(AssetId id, NoSettings settings, ref SequenceReader reader); public abstract Task Encode(AssetId id, NoSettings settings, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer); - public abstract void HotSwap(TAsset instance, TAsset replacement); + public abstract void HotSwap(TAsset instance, TAsset newParts); public NoSettings ReadSettings(ref SequenceReader reader) { diff --git a/source/CapriKit.DirectX11/Resources/Shaders/VertexShader.cs b/source/CapriKit.DirectX11/Resources/Shaders/VertexShader.cs index 31e9fa4..c43bf30 100644 --- a/source/CapriKit.DirectX11/Resources/Shaders/VertexShader.cs +++ b/source/CapriKit.DirectX11/Resources/Shaders/VertexShader.cs @@ -9,12 +9,12 @@ public interface IVertexShader : IDisposable IInputLayout CreateInputLayout(Device device, InputElementDescription[] elements); - public void HotSwap(IVertexShader replacement) + public void HotSwap(IVertexShader newParts) { - Blob = replacement.Blob; + Blob = newParts.Blob; var oldShader = ID3D11VertexShader; - ID3D11VertexShader = replacement.ID3D11VertexShader; + ID3D11VertexShader = newParts.ID3D11VertexShader; oldShader.Dispose(); } } diff --git a/source/CapriKit.Tests/DirectX11/Buffers/GenericBufferTests.cs b/source/CapriKit.Tests/DirectX11/Buffers/GenericBufferTests.cs index ec6f046..82320c2 100644 --- a/source/CapriKit.Tests/DirectX11/Buffers/GenericBufferTests.cs +++ b/source/CapriKit.Tests/DirectX11/Buffers/GenericBufferTests.cs @@ -69,7 +69,8 @@ public async Task Mix_Upload_Modify_Download_Staging() private static IComputeShader Create(Device device) { var fileSystem = new InMemoryFileSystem(); - return ShaderCompiler.CompileComputeShader(fileSystem, string.Empty, device, ShaderSource, "CS", "DeviceTest.cs"); + var includePath = new DirectoryPath(Path.GetTempPath()); + return ShaderCompiler.CompileComputeShader(fileSystem, includePath, device, ShaderSource, "CS", "DeviceTest.cs"); } private const string ShaderSource = """ From 8a92865fc876e26b3c857a6377fefe5ef554af6e Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Fri, 24 Jul 2026 15:57:24 +0200 Subject: [PATCH 19/53] Move to zero physical IO --- CapriKit.slnx | 1 + Directory.Packages.props | 3 +- PLANNING.md | 12 ++++ source/CapriKit.AssetPipeline/Asset.cs | 7 ++ source/CapriKit.AssetPipeline/AssetId.cs | 5 ++ source/CapriKit.AssetPipeline/AssetManager.cs | 23 +++++-- .../CapriKit.AssetPipeline.csproj | 1 + .../HotReloading/HotSwapManager.cs | 16 +++-- .../ServiceCollectionExtensions.cs | 18 +++++ source/CapriKit.IO/FileSystem.cs | 7 ++ source/CapriKit.IO/FileSystemEventQueue.cs | 35 ---------- source/CapriKit.IO/IVirtualFileSystem.cs | 7 ++ source/CapriKit.IO/InMemoryFileSystem.cs | 69 +++++++++++++++++-- .../ReadOnlyVirtualFileSystemSpy.cs | 7 ++ source/CapriKit.IO/ScopedFileSystem.cs | 29 +++++++- source/CapriKit.IO/VirtualFileSystemSpy.cs | 7 ++ .../{ => Watchers}/FileSystemEventListener.cs | 32 ++++----- .../Watchers/FileSystemEventQueue.cs | 25 +++++++ .../Watchers/IVirtualFileSystemWatcher.cs | 21 ++++++ .../Watchers/ScopedFileSystemEventListener.cs | 26 +++++++ .../AssetPipeline/AssetManagerTests.cs | 7 +- .../FileSystemEventListenerTests.cs | 11 ++- .../TestUtilities/FileSystemUtilities.cs | 17 +++++ 23 files changed, 301 insertions(+), 85 deletions(-) create mode 100644 PLANNING.md create mode 100644 source/CapriKit.AssetPipeline/ServiceCollectionExtensions.cs delete mode 100644 source/CapriKit.IO/FileSystemEventQueue.cs rename source/CapriKit.IO/{ => Watchers}/FileSystemEventListener.cs (58%) create mode 100644 source/CapriKit.IO/Watchers/FileSystemEventQueue.cs create mode 100644 source/CapriKit.IO/Watchers/IVirtualFileSystemWatcher.cs create mode 100644 source/CapriKit.IO/Watchers/ScopedFileSystemEventListener.cs rename source/CapriKit.Tests/IO/{ => Watchers}/FileSystemEventListenerTests.cs (84%) create mode 100644 source/CapriKit.Tests/TestUtilities/FileSystemUtilities.cs diff --git a/CapriKit.slnx b/CapriKit.slnx index 4c79f7b..7ec1e69 100644 --- a/CapriKit.slnx +++ b/CapriKit.slnx @@ -18,6 +18,7 @@ + diff --git a/Directory.Packages.props b/Directory.Packages.props index e52ace0..5620115 100644 --- a/Directory.Packages.props +++ b/Directory.Packages.props @@ -13,6 +13,7 @@ + @@ -22,4 +23,4 @@ - \ No newline at end of file + diff --git a/PLANNING.md b/PLANNING.md new file mode 100644 index 0000000..9ef3b95 --- /dev/null +++ b/PLANNING.md @@ -0,0 +1,12 @@ +# Planning +_Ideas and future plans for the projects in this repository_ + +## Soon +- Finish integrating hot-reloading in the asset pipeline +- Test hot reloading in the live test project +- Clean up an asset-pipeline code and merge branch + +## Later +- Add logging support via `Microsoft.Extensions.Logging.Abstractions` to most projects +- Add dependency injection support via `Microsoft.Extensions.DependencyInjection.Abstractions` to most projects and use a LightInject extension/implementation in the demo project +- Add metrics support (if needed) via `System.Diagnostics`: `.Activity`, `.Meter`, `.Counter`, which are Open-Telemetry compatible. diff --git a/source/CapriKit.AssetPipeline/Asset.cs b/source/CapriKit.AssetPipeline/Asset.cs index cc805d8..0008a22 100644 --- a/source/CapriKit.AssetPipeline/Asset.cs +++ b/source/CapriKit.AssetPipeline/Asset.cs @@ -1,4 +1,11 @@ namespace CapriKit.AssetPipeline; +/// +/// An active asset +/// +/// +/// The unique id, refers to a virtual file location (id.Path) and if required a sub-resource in that file (id.Key). +/// The asset +/// Files that this asset depends on, if any of these files changed the asset needs te be rebuild. public sealed record Asset(AssetId Id, T Value, IReadOnlyList Dependencies) where T : class; diff --git a/source/CapriKit.AssetPipeline/AssetId.cs b/source/CapriKit.AssetPipeline/AssetId.cs index d41a2ec..c98b808 100644 --- a/source/CapriKit.AssetPipeline/AssetId.cs +++ b/source/CapriKit.AssetPipeline/AssetId.cs @@ -2,4 +2,9 @@ namespace CapriKit.AssetPipeline; +/// +/// Unique asset identifier +/// +/// Optional key to a sub-resources in Path. +/// Virtual file path that points to the file the asset originates from. public record AssetId(string Key, FilePath Path); diff --git a/source/CapriKit.AssetPipeline/AssetManager.cs b/source/CapriKit.AssetPipeline/AssetManager.cs index 31b7ff6..f62ec91 100644 --- a/source/CapriKit.AssetPipeline/AssetManager.cs +++ b/source/CapriKit.AssetPipeline/AssetManager.cs @@ -1,26 +1,34 @@ +using CapriKit.AssetPipeline.HotReloading; using CapriKit.IO; +using Microsoft.Extensions.Logging; using static CapriKit.AssetPipeline.AssetUtilities; namespace CapriKit.AssetPipeline; +/// +/// Encodes, decodes, loads and tracks assets. +/// public sealed class AssetManager { private readonly IVirtualFileSystem FileSystem; - - // Values are IAssetTranscoder instances, keyed by typeof(TAsset) private readonly Dictionary Transcoders = []; private readonly AssetCache Cache; + private readonly HotSwapManager HotSwapManager; - public AssetManager(DirectoryPath rootDirectory) - : this(new FileSystem().ScopedTo(rootDirectory)) { } + public AssetManager(ILoggerFactory logger, DirectoryPath rootDirectory) + : this(logger, new FileSystem().ScopedTo(rootDirectory)) { } - public AssetManager(IVirtualFileSystem fileSystem) + public AssetManager(ILoggerFactory logger, IVirtualFileSystem fileSystem) { FileSystem = fileSystem; Cache = new AssetCache(); + HotSwapManager = new HotSwapManager(logger, this, fileSystem); } + /// public void PushScope() => Cache.PushScope(); + + /// public void PopScope() => Cache.PopScope(); public void RegisterTranscoder(IAssetTranscoder transcoder) @@ -39,6 +47,9 @@ public Task Encode(AssetId id) return Encode(id, default(NoSettings)); } + /// + /// Immediately decodes an asset, bypasses cache and hot reloading mechanisms. + /// public async Task Decode(AssetId id) where TAsset : class { @@ -67,7 +78,7 @@ public async Task Load(AssetId id, IAssetSettings settin } var asset = await DecodeOrBuild(id, settings); - Cache.Add(id, asset.Value); + Cache.Add(id, asset.Value); return asset.Value; } diff --git a/source/CapriKit.AssetPipeline/CapriKit.AssetPipeline.csproj b/source/CapriKit.AssetPipeline/CapriKit.AssetPipeline.csproj index f1cb5b1..3c2e557 100644 --- a/source/CapriKit.AssetPipeline/CapriKit.AssetPipeline.csproj +++ b/source/CapriKit.AssetPipeline/CapriKit.AssetPipeline.csproj @@ -1,6 +1,7 @@  + diff --git a/source/CapriKit.AssetPipeline/HotReloading/HotSwapManager.cs b/source/CapriKit.AssetPipeline/HotReloading/HotSwapManager.cs index e2bdb6c..80bbd24 100644 --- a/source/CapriKit.AssetPipeline/HotReloading/HotSwapManager.cs +++ b/source/CapriKit.AssetPipeline/HotReloading/HotSwapManager.cs @@ -1,5 +1,6 @@ using CapriKit.Concurrency.Async; using CapriKit.IO; +using CapriKit.IO.Watchers; using Microsoft.Extensions.Logging; using System.Collections.Concurrent; using System.Diagnostics; @@ -18,26 +19,28 @@ internal sealed partial class HotSwapManager : IDisposable private readonly ILogger Logger; private readonly AssetManager AssetManager; - private readonly ReadOnlyScopedFileSystem FileSystem; + private readonly IReadOnlyVirtualFileSystem FileSystem; private readonly Dictionary Tracked; private readonly Dictionary> Dependents; + private readonly IVirtualFileSystemWatcher Watcher; private readonly FileSystemEventQueue PendingFileChanges; private readonly HashSet PendingReloads; private readonly ConcurrentQueue PendingHotSwaps; private long lastFileChange; - public HotSwapManager(ILogger logger, AssetManager assetManager, ReadOnlyScopedFileSystem fileSystem) + public HotSwapManager(ILoggerFactory logger, AssetManager assetManager, IReadOnlyVirtualFileSystem fileSystem) { - Logger = logger; + Logger = logger.CreateLogger(); AssetManager = assetManager; FileSystem = fileSystem; Tracked = []; Dependents = []; - PendingFileChanges = new FileSystemEventQueue(fileSystem.BasePath); + Watcher = fileSystem.Watch(DirectoryPath.Empty); // Watch for all changes, usually fileSystem is a ScopedVirtualFileSystem + PendingFileChanges = new FileSystemEventQueue(Watcher); PendingReloads = []; PendingHotSwaps = []; lastFileChange = Stopwatch.GetTimestamp(); @@ -49,8 +52,7 @@ public void Track(Asset asset, IAssetSettings settings) Tracked[asset.Id] = new Reloadable(asset.Id, asset.Value, settings); foreach (var dependency in asset.Dependencies) { - // Match the absolute paths generated by file system events - var file = FileSystem.GetAbsolutePath(dependency.File); + var file = dependency.File; if (!Dependents.TryGetValue(file, out var ids)) { ids = Dependents[file] = []; } ids.Add(asset.Id); } @@ -111,7 +113,7 @@ private void HotSwapCompleted() public void Dispose() { - PendingFileChanges.Dispose(); + Watcher.Stop(); } [LoggerMessage(Level = LogLevel.Information, Message = "Detected file change: {path}, affecting asset: {asset}")] diff --git a/source/CapriKit.AssetPipeline/ServiceCollectionExtensions.cs b/source/CapriKit.AssetPipeline/ServiceCollectionExtensions.cs new file mode 100644 index 0000000..ab51c87 --- /dev/null +++ b/source/CapriKit.AssetPipeline/ServiceCollectionExtensions.cs @@ -0,0 +1,18 @@ +using CapriKit.IO; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging; + +namespace CapriKit.AssetPipeline; + +// For Microsoft.Extensions.DependencyInjection.Abstractions +public static class ServiceCollectionExtensions +{ + public static IServiceCollection AddAssetPipeline(this IServiceCollection services, DirectoryPath assetDirectory) + { + return services.AddSingleton(sp => + { + var logFactory = sp.GetRequiredService(); + return new AssetManager(logFactory, assetDirectory); + }); + } +} diff --git a/source/CapriKit.IO/FileSystem.cs b/source/CapriKit.IO/FileSystem.cs index 750e919..932290f 100644 --- a/source/CapriKit.IO/FileSystem.cs +++ b/source/CapriKit.IO/FileSystem.cs @@ -1,3 +1,5 @@ +using CapriKit.IO.Watchers; + namespace CapriKit.IO; public class FileSystem : IVirtualFileSystem @@ -85,6 +87,11 @@ public IReadOnlyList List(DirectoryPath directory) return filePaths; } + public IVirtualFileSystemWatcher Watch(DirectoryPath directory, bool includeSubDirectories = true) + { + return new FileSystemEventListener(directory, includeSubDirectories); + } + private FileInfo FindOrThrow(FilePath file) { var info = GetFileInfo(file); diff --git a/source/CapriKit.IO/FileSystemEventQueue.cs b/source/CapriKit.IO/FileSystemEventQueue.cs deleted file mode 100644 index 2e59fbe..0000000 --- a/source/CapriKit.IO/FileSystemEventQueue.cs +++ /dev/null @@ -1,35 +0,0 @@ -using System.Collections.Concurrent; -using System.Diagnostics.CodeAnalysis; - -namespace CapriKit.IO; - -/// -/// Listens for file changes and puts events in a queue so an interested party can control when to handle them. -/// -public sealed class FileSystemEventQueue : IDisposable -{ - private readonly ConcurrentQueue Queue; - private readonly FileSystemEventListener Events; - - public FileSystemEventQueue(DirectoryPath directory, bool includeSubDirectories = true) - { - Queue = new ConcurrentQueue(); - Events = new FileSystemEventListener(directory, includeSubDirectories); - Events.OnFileChanged += (s, e) => - { - Queue.Enqueue(e); - }; - } - - public int Count => Queue.Count; - - public bool TryDequeue([NotNullWhen(true)] out FileSystemEvent? @event) - { - return Queue.TryDequeue(out @event); - } - - public void Dispose() - { - Events.Dispose(); - } -} diff --git a/source/CapriKit.IO/IVirtualFileSystem.cs b/source/CapriKit.IO/IVirtualFileSystem.cs index f1270f6..9ae8b6d 100644 --- a/source/CapriKit.IO/IVirtualFileSystem.cs +++ b/source/CapriKit.IO/IVirtualFileSystem.cs @@ -1,3 +1,5 @@ +using CapriKit.IO.Watchers; + namespace CapriKit.IO; public interface IVirtualFileSystem : IReadOnlyVirtualFileSystem @@ -44,4 +46,9 @@ public interface IReadOnlyVirtualFileSystem /// Lists all files in the given directory. Throws an exception if the directory does not exist. /// IReadOnlyList List(DirectoryPath directory); + + /// + /// Watches for changes in the given subdirectory. + /// + IVirtualFileSystemWatcher Watch(DirectoryPath directory, bool includeSubDirectories = true); } diff --git a/source/CapriKit.IO/InMemoryFileSystem.cs b/source/CapriKit.IO/InMemoryFileSystem.cs index 6dc1940..9a2ab68 100644 --- a/source/CapriKit.IO/InMemoryFileSystem.cs +++ b/source/CapriKit.IO/InMemoryFileSystem.cs @@ -1,13 +1,17 @@ +using CapriKit.IO.Watchers; + namespace CapriKit.IO; public sealed class InMemoryFileSystem : IVirtualFileSystem { private record InMemoryFile(InMemoryFileStream Stream, DateTime LastWriteTime); private readonly Dictionary Disk; + private readonly List Watchers; public InMemoryFileSystem() { - Disk = new Dictionary(); + Disk = []; + Watchers = []; } public Stream AppendWrite(FilePath file) @@ -17,6 +21,7 @@ public Stream AppendWrite(FilePath file) var stream = inMemoryFile.Stream; stream.Position = stream.Length; + RaiseChange(file, FileSystemChangeKind.Changed); return stream; } @@ -27,17 +32,22 @@ public Stream CreateReadWrite(FilePath file) Disk[file] = inMemoryFile with { LastWriteTime = DateTime.Now }; inMemoryFile.Stream.SetLength(0); inMemoryFile.Stream.Position = 0; + RaiseChange(file, FileSystemChangeKind.Changed); return inMemoryFile.Stream; } var newStream = new InMemoryFileStream(); Disk.Add(file, new InMemoryFile(newStream, DateTime.Now)); + RaiseChange(file, FileSystemChangeKind.Created); return newStream; } public void Delete(FilePath file) { - Disk.Remove(file); + if (Disk.Remove(file)) + { + RaiseChange(file, FileSystemChangeKind.Deleted); + } } public bool Exists(FilePath file) @@ -78,6 +88,13 @@ public IReadOnlyList List(DirectoryPath directory) return files; } + public IVirtualFileSystemWatcher Watch(DirectoryPath directory, bool includeSubDirectories = true) + { + var watcher = new InMemoryFileSystemWatcher(this, directory, includeSubDirectories); + Watchers.Add(watcher); + return watcher; + } + private InMemoryFile FindOrThrow(FilePath file) { if (Disk.TryGetValue(file, out var value)) @@ -87,22 +104,60 @@ private InMemoryFile FindOrThrow(FilePath file) throw new FileNotFoundException(null, file.ToString()); } + private void RaiseChange(FilePath file, FileSystemChangeKind kind) + { + var @event = new VirtualFileSystemEvent(file, kind); + foreach (var watcher in Watchers.ToArray()) + { + watcher.Notify(this, @event); + } + } + + private void RemoveWatcher(InMemoryFileSystemWatcher watcher) + { + Watchers.Remove(watcher); + } + private sealed class InMemoryFileStream : MemoryStream { + // Prevent disposing of the actual stream, just rewind it protected override void Dispose(bool disposing) { Position = 0; } + } + + private sealed class InMemoryFileSystemWatcher( + InMemoryFileSystem owner, DirectoryPath directory, bool includeSubDirectories) + : IVirtualFileSystemWatcher + { + private event VirtualFileSystemEventHandler? onFileChanged; - public override ValueTask DisposeAsync() + public event VirtualFileSystemEventHandler? OnFileChanged { - Position = 0; - return ValueTask.CompletedTask; + add => onFileChanged += value; + remove => onFileChanged -= value; } - public override void Close() + internal void Notify(object sender, VirtualFileSystemEvent @event) { - Position = 0; + if (Matches(@event.File)) + { + onFileChanged?.Invoke(sender, @event); + } + } + + private bool Matches(FilePath file) + { + var fileDirectory = file.Directory; + return includeSubDirectories + ? fileDirectory.StartsWith(directory) + : fileDirectory.Equals(directory); + } + + public void Stop() + { + owner.RemoveWatcher(this); } } } diff --git a/source/CapriKit.IO/ReadOnlyVirtualFileSystemSpy.cs b/source/CapriKit.IO/ReadOnlyVirtualFileSystemSpy.cs index 836062a..bc3f910 100644 --- a/source/CapriKit.IO/ReadOnlyVirtualFileSystemSpy.cs +++ b/source/CapriKit.IO/ReadOnlyVirtualFileSystemSpy.cs @@ -1,3 +1,5 @@ +using CapriKit.IO.Watchers; + namespace CapriKit.IO; /// @@ -44,4 +46,9 @@ public long SizeInBytes(FilePath file) { return Actual.SizeInBytes(file); } + + public IVirtualFileSystemWatcher Watch(DirectoryPath directory, bool includeSubDirectories = true) + { + return Actual.Watch(directory, includeSubDirectories); + } } diff --git a/source/CapriKit.IO/ScopedFileSystem.cs b/source/CapriKit.IO/ScopedFileSystem.cs index 594c5a4..f909e65 100644 --- a/source/CapriKit.IO/ScopedFileSystem.cs +++ b/source/CapriKit.IO/ScopedFileSystem.cs @@ -1,3 +1,4 @@ +using CapriKit.IO.Watchers; using System.Diagnostics; namespace CapriKit.IO; @@ -57,7 +58,7 @@ public ReadOnlyScopedFileSystem(IReadOnlyVirtualFileSystem source, DirectoryPath /// - /// Gets the absolute path of a directory contained in this scoped file system. To be used with IO methods that are + /// Gets the full path of a directory contained in this scoped file system. To be used with IO methods that are /// not aware of the virtual file system. Throws if the directory points to outside the scoped file system, /// public DirectoryPath GetAbsolutePath(DirectoryPath directory) => GetDirectoryPath(directory); @@ -95,6 +96,12 @@ public long SizeInBytes(FilePath file) protected DirectoryPath GetDirectoryPath(DirectoryPath path) { + if (path.IsAbsolute) + { + ThrowIfPathIsOutsideBasePath(path); + return path; + } + var fullPath = path.GetPathRelativeTo(BasePath); ThrowIfPathIsOutsideBasePath(fullPath); @@ -103,6 +110,12 @@ protected DirectoryPath GetDirectoryPath(DirectoryPath path) protected FilePath GetFilePath(FilePath path) { + if (path.IsAbsolute) + { + ThrowIfPathIsOutsideBasePath(path); + return path; + } + var fullPath = path.GetPathRelativeTo(BasePath); ThrowIfPathIsOutsideBasePath(fullPath); @@ -126,4 +139,18 @@ protected void ThrowIfPathIsOutsideBasePath(DirectoryPath path) throw new ForbiddenPathException(path, BasePath); } } + + public IVirtualFileSystemWatcher Watch(DirectoryPath directory, bool includeSubDirectories = true) + { + ThrowIfPathIsOutsideBasePath(directory); + var fullPath = GetAbsolutePath(directory); + var watchers = Source.Watch(fullPath, includeSubDirectories); + return new ScopedFileSystemEventListener(watchers, BasePath); + } + + public IVirtualFileSystemWatcher Watch(bool includeSubDirectories = true) + { + var watchers = Source.Watch(BasePath, includeSubDirectories); + return new ScopedFileSystemEventListener(watchers, BasePath); + } } diff --git a/source/CapriKit.IO/VirtualFileSystemSpy.cs b/source/CapriKit.IO/VirtualFileSystemSpy.cs index a9ed7a0..e02bdc9 100644 --- a/source/CapriKit.IO/VirtualFileSystemSpy.cs +++ b/source/CapriKit.IO/VirtualFileSystemSpy.cs @@ -1,3 +1,5 @@ +using CapriKit.IO.Watchers; + namespace CapriKit.IO; /// @@ -64,4 +66,9 @@ public long SizeInBytes(FilePath file) { return Actual.SizeInBytes(file); } + + public IVirtualFileSystemWatcher Watch(DirectoryPath directory, bool includeSubDirectories = true) + { + return Actual.Watch(directory, includeSubDirectories); + } } diff --git a/source/CapriKit.IO/FileSystemEventListener.cs b/source/CapriKit.IO/Watchers/FileSystemEventListener.cs similarity index 58% rename from source/CapriKit.IO/FileSystemEventListener.cs rename to source/CapriKit.IO/Watchers/FileSystemEventListener.cs index df8d681..a160f89 100644 --- a/source/CapriKit.IO/FileSystemEventListener.cs +++ b/source/CapriKit.IO/Watchers/FileSystemEventListener.cs @@ -1,26 +1,13 @@ -namespace CapriKit.IO; - -public enum FileSystemChangeKind -{ - Created, - Changed, - Deleted, -} - -/// The absolute path to the file affected -/// The kind of change the file underwent -public record FileSystemEvent(FilePath File, FileSystemChangeKind Kind); - -public delegate void FileSystemEventHandler(object sender, FileSystemEvent e); +namespace CapriKit.IO.Watchers; /// /// Listens for file changes and notifies interested parties via an event /// -public sealed class FileSystemEventListener : IDisposable +public sealed class FileSystemEventListener : IVirtualFileSystemWatcher, IDisposable { private readonly FileSystemWatcher Watcher; - private event FileSystemEventHandler? onFileChanged; + private event VirtualFileSystemEventHandler? onFileChanged; public FileSystemEventListener(DirectoryPath directory, bool includeSubDirectories = true) { @@ -38,12 +25,12 @@ public FileSystemEventListener(DirectoryPath directory, bool includeSubDirectori EnableRaisingEvents = true, }; - Watcher.Created += (s, e) => onFileChanged?.Invoke(s, new FileSystemEvent(FileSystem.GetFilePath(e.FullPath), FileSystemChangeKind.Created)); - Watcher.Changed += (s, e) => onFileChanged?.Invoke(s, new FileSystemEvent(FileSystem.GetFilePath(e.FullPath), FileSystemChangeKind.Changed)); - Watcher.Deleted += (s, e) => onFileChanged?.Invoke(s, new FileSystemEvent(FileSystem.GetFilePath(e.FullPath), FileSystemChangeKind.Deleted)); + Watcher.Created += (s, e) => onFileChanged?.Invoke(s, new VirtualFileSystemEvent(FileSystem.GetFilePath(e.FullPath), FileSystemChangeKind.Created)); + Watcher.Changed += (s, e) => onFileChanged?.Invoke(s, new VirtualFileSystemEvent(FileSystem.GetFilePath(e.FullPath), FileSystemChangeKind.Changed)); + Watcher.Deleted += (s, e) => onFileChanged?.Invoke(s, new VirtualFileSystemEvent(FileSystem.GetFilePath(e.FullPath), FileSystemChangeKind.Deleted)); } - public event FileSystemEventHandler? OnFileChanged + public event VirtualFileSystemEventHandler? OnFileChanged { add { @@ -57,6 +44,11 @@ public event FileSystemEventHandler? OnFileChanged public DirectoryPath Directory { get; } + public void Stop() + { + Dispose(); + } + public void Dispose() { Watcher.Dispose(); diff --git a/source/CapriKit.IO/Watchers/FileSystemEventQueue.cs b/source/CapriKit.IO/Watchers/FileSystemEventQueue.cs new file mode 100644 index 0000000..0f1e537 --- /dev/null +++ b/source/CapriKit.IO/Watchers/FileSystemEventQueue.cs @@ -0,0 +1,25 @@ +using System.Collections.Concurrent; +using System.Diagnostics.CodeAnalysis; + +namespace CapriKit.IO.Watchers; + +/// +/// Listens for file changes and puts events in a queue so an interested party can control when to handle them. +/// +public sealed class FileSystemEventQueue +{ + private readonly ConcurrentQueue Queue; + + public FileSystemEventQueue(IVirtualFileSystemWatcher watcher) + { + Queue = new ConcurrentQueue(); + watcher.OnFileChanged += (s, e) => Queue.Enqueue(e); + } + + public int Count => Queue.Count; + + public bool TryDequeue([NotNullWhen(true)] out VirtualFileSystemEvent? @event) + { + return Queue.TryDequeue(out @event); + } +} diff --git a/source/CapriKit.IO/Watchers/IVirtualFileSystemWatcher.cs b/source/CapriKit.IO/Watchers/IVirtualFileSystemWatcher.cs new file mode 100644 index 0000000..4602d81 --- /dev/null +++ b/source/CapriKit.IO/Watchers/IVirtualFileSystemWatcher.cs @@ -0,0 +1,21 @@ +namespace CapriKit.IO.Watchers; + +public enum FileSystemChangeKind +{ + Created, + Changed, + Deleted, +} + +/// The absolute path to the file affected +/// The kind of change the file underwent +public record VirtualFileSystemEvent(FilePath File, FileSystemChangeKind Kind); + +public delegate void VirtualFileSystemEventHandler(object sender, VirtualFileSystemEvent e); + +public interface IVirtualFileSystemWatcher +{ + public event VirtualFileSystemEventHandler? OnFileChanged; + + public void Stop(); +} diff --git a/source/CapriKit.IO/Watchers/ScopedFileSystemEventListener.cs b/source/CapriKit.IO/Watchers/ScopedFileSystemEventListener.cs new file mode 100644 index 0000000..61fbd46 --- /dev/null +++ b/source/CapriKit.IO/Watchers/ScopedFileSystemEventListener.cs @@ -0,0 +1,26 @@ +namespace CapriKit.IO.Watchers; + +/// +/// Wrapper for a general IVirtualFileSystemWatcher that ensures paths are relative to the basePath +/// +internal class ScopedFileSystemEventListener : IVirtualFileSystemWatcher +{ + private readonly IVirtualFileSystemWatcher Inner; + private readonly DirectoryPath BasePath; + + public ScopedFileSystemEventListener(IVirtualFileSystemWatcher inner, DirectoryPath basePath) + { + Inner = inner; + BasePath = basePath; + Inner.OnFileChanged += OnInner; + } + + public event VirtualFileSystemEventHandler? OnFileChanged; + + private void OnInner(object sender, VirtualFileSystemEvent e) + { + OnFileChanged?.Invoke(this, e with { File = e.File.GetPathRelativeTo(BasePath) }); + } + + public void Stop() { Inner.OnFileChanged -= OnInner; Inner.Stop(); } +} diff --git a/source/CapriKit.Tests/AssetPipeline/AssetManagerTests.cs b/source/CapriKit.Tests/AssetPipeline/AssetManagerTests.cs index e1fb0bc..e89985b 100644 --- a/source/CapriKit.Tests/AssetPipeline/AssetManagerTests.cs +++ b/source/CapriKit.Tests/AssetPipeline/AssetManagerTests.cs @@ -1,5 +1,6 @@ using CapriKit.AssetPipeline; using CapriKit.IO; +using Microsoft.Extensions.Logging.Abstractions; namespace CapriKit.Tests.AssetPipeline; @@ -9,8 +10,9 @@ internal class AssetManagerTests public async Task Decode() { var fileSystem = new InMemoryFileSystem(); + await fileSystem.WriteAllText("hello.txt", "héllo"); - var manager = new AssetManager(fileSystem); + var manager = new AssetManager(NullLoggerFactory.Instance, fileSystem); manager.RegisterTranscoder(new DummyTranscoder()); var id = new AssetId("Main", "hello.txt"); @@ -24,8 +26,9 @@ public async Task Decode() public async Task Decode_SettingsAreReadFromTheEncodedFile() { var fileSystem = new InMemoryFileSystem(); + await fileSystem.WriteAllText("hello.txt", "hey"); - var manager = new AssetManager(fileSystem); + var manager = new AssetManager(NullLoggerFactory.Instance, fileSystem); manager.RegisterTranscoder(new RepeatTranscoder()); var id = new AssetId("Main", "hello.txt"); diff --git a/source/CapriKit.Tests/IO/FileSystemEventListenerTests.cs b/source/CapriKit.Tests/IO/Watchers/FileSystemEventListenerTests.cs similarity index 84% rename from source/CapriKit.Tests/IO/FileSystemEventListenerTests.cs rename to source/CapriKit.Tests/IO/Watchers/FileSystemEventListenerTests.cs index 5cd4859..6503718 100644 --- a/source/CapriKit.Tests/IO/FileSystemEventListenerTests.cs +++ b/source/CapriKit.Tests/IO/Watchers/FileSystemEventListenerTests.cs @@ -1,6 +1,8 @@ using CapriKit.IO; +using CapriKit.IO.Watchers; +using CapriKit.Tests.TestUtilities; -namespace CapriKit.Tests.IO; +namespace CapriKit.Tests.IO.Watchers; internal class FileSystemEventListenerTests { @@ -9,10 +11,7 @@ internal class FileSystemEventListenerTests [Before(Test)] public void TestSetup() { - var id = Path.GetRandomFileName(); - var path = Path.Combine(Path.GetTempPath(), $"{nameof(FileSystemEventListenerTests)}.{id}"); - TempDirectory = new DirectoryPath(path); - Directory.CreateDirectory(TempDirectory); + TempDirectory = FileSystemUtilities.CreateTemporaryDirectory(); } [After(Test)] @@ -32,7 +31,7 @@ public async Task OnFileChanged() var fileSystem = new FileSystem(); var scopedFileSystem = new ScopedFileSystem(fileSystem, TempDirectory); - using var watcher = new FileSystemEventListener(TempDirectory, false); + var watcher = scopedFileSystem.Watch(TempDirectory, false); watcher.OnFileChanged += (s, e) => { if (e.Kind == FileSystemChangeKind.Created && e.File.FileName.Equals(fileName, StringComparison.OrdinalIgnoreCase)) diff --git a/source/CapriKit.Tests/TestUtilities/FileSystemUtilities.cs b/source/CapriKit.Tests/TestUtilities/FileSystemUtilities.cs new file mode 100644 index 0000000..85db355 --- /dev/null +++ b/source/CapriKit.Tests/TestUtilities/FileSystemUtilities.cs @@ -0,0 +1,17 @@ +using CapriKit.IO; +using CapriKit.Tests.IO.Watchers; + +namespace CapriKit.Tests.TestUtilities; + +internal static class FileSystemUtilities +{ + public static DirectoryPath CreateTemporaryDirectory() + { + var id = Path.GetRandomFileName(); + var path = Path.Combine(Path.GetTempPath(), $"{nameof(FileSystemEventListenerTests)}.{id}"); + var directory = new DirectoryPath(path); + Directory.CreateDirectory(directory); + + return directory; + } +} From ee28cc2e86a390826c224c7aeda6428fc8a7ee90 Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Sun, 26 Jul 2026 21:29:16 +0200 Subject: [PATCH 20/53] Remove exception throwing from decoding --- source/CapriKit.AssetPipeline/Asset.cs | 91 +++++++++++++++- source/CapriKit.AssetPipeline/AssetDecoder.cs | 15 ++- .../CapriKit.AssetPipeline/AssetFileCache.cs | 58 ++++++++++ source/CapriKit.AssetPipeline/AssetId.cs | 10 -- source/CapriKit.AssetPipeline/AssetManager.cs | 100 +++++++----------- .../{AssetCache.cs => AssetMemoryCache.cs} | 4 +- .../CapriKit.AssetPipeline/AssetUtilities.cs | 20 ---- source/CapriKit.AssetPipeline/Dependency.cs | 5 - .../HotReloading/HotSwapManager.cs | 11 +- .../HotReloading/HotSwappable.cs | 20 +++- .../HotReloading/Reloadable.cs | 2 +- .../TranscoderCollection.cs | 21 ++++ .../AssetPipeline/AssetDecoderTests.cs | 14 ++- 13 files changed, 256 insertions(+), 115 deletions(-) create mode 100644 source/CapriKit.AssetPipeline/AssetFileCache.cs delete mode 100644 source/CapriKit.AssetPipeline/AssetId.cs rename source/CapriKit.AssetPipeline/{AssetCache.cs => AssetMemoryCache.cs} (92%) delete mode 100644 source/CapriKit.AssetPipeline/Dependency.cs create mode 100644 source/CapriKit.AssetPipeline/TranscoderCollection.cs diff --git a/source/CapriKit.AssetPipeline/Asset.cs b/source/CapriKit.AssetPipeline/Asset.cs index 0008a22..45f5c6f 100644 --- a/source/CapriKit.AssetPipeline/Asset.cs +++ b/source/CapriKit.AssetPipeline/Asset.cs @@ -1,11 +1,96 @@ +using CapriKit.IO; +using System.Diagnostics.CodeAnalysis; +using System.Runtime.ExceptionServices; + namespace CapriKit.AssetPipeline; +/// +/// Unique asset identifier +/// +/// Optional key to a sub-resources in Path. +/// Virtual file path that points to the file the asset originates from. +public record AssetId(string Key, FilePath Path); + + /// /// An active asset /// -/// +/// /// The unique id, refers to a virtual file location (id.Path) and if required a sub-resource in that file (id.Key). /// The asset +/// The settings used for encoding/decoding /// Files that this asset depends on, if any of these files changed the asset needs te be rebuild. -public sealed record Asset(AssetId Id, T Value, IReadOnlyList Dependencies) - where T : class; +public sealed record Asset(AssetId Id, TAsset Value, IAssetSettings Settings, IReadOnlyList Dependencies) + where TAsset : class; + +public sealed record Dependency(FilePath File, DateTime Version); + + +/// +/// Result of building an asset +/// +public sealed class AssetJob + where TAsset : class + +{ + private readonly Asset? Asset; + private readonly ExceptionDispatchInfo? Exception; + + private AssetJob(AssetId id, Asset? asset, ExceptionDispatchInfo? exception) + { + Id = id; + Asset = asset; + Exception = exception; + } + + public AssetId Id { get; } + + public bool OnSuccess([NotNullWhen(true)] out Asset? asset) + { + asset = Asset; + return asset != null; + } + + public bool OnFailure([NotNullWhen(true)] out ExceptionDispatchInfo? exception) + { + exception = Exception; + return exception != null; + } + + public bool OnMissing() + { + return Asset == null && Exception == null; + } + + public static AssetJob Failure(AssetId id, ExceptionDispatchInfo exception) + { + return new AssetJob(id, null, exception); + } + + public static AssetJob Success(AssetId id, Asset asset) + { + return new AssetJob(id, asset, null); + } + + public static AssetJob Missing(AssetId id) + { + return new AssetJob(id, null, null); + } + + public void Match(Action> onSuccess, Action onFailure, Action onMissing) + { + if (Asset != null) { onSuccess(Id, Asset); } + else if (Exception != null) { onFailure(Id, Exception); } + else { onMissing(Id); } + } + + public TReturn Match(Func, TReturn> onSuccess, + Func onFailure, + Func onMissing) + { + if (Asset != null) { return onSuccess(Id, Asset); } + if (Exception != null) { return onFailure(Id, Exception); } + return onMissing(Id); + } +} + diff --git a/source/CapriKit.AssetPipeline/AssetDecoder.cs b/source/CapriKit.AssetPipeline/AssetDecoder.cs index 710accf..db428b2 100644 --- a/source/CapriKit.AssetPipeline/AssetDecoder.cs +++ b/source/CapriKit.AssetPipeline/AssetDecoder.cs @@ -1,6 +1,7 @@ using CapriKit.IO; using CapriKit.IO.Streams; using System.Buffers; +using System.Runtime.ExceptionServices; using static CapriKit.AssetPipeline.AssetUtilities; namespace CapriKit.AssetPipeline; @@ -10,12 +11,15 @@ namespace CapriKit.AssetPipeline; /// internal static class AssetDecoder { - public static async Task> Decode(AssetId id, + public static async Task> Decode(AssetId id, IAssetTranscoder decoder, IVirtualFileSystem fileSystem) where TAsset : class { var inputPath = ToEncodedFilePath(id); - ThrowOnFileNotFound(inputPath, fileSystem); + if (!fileSystem.Exists(inputPath)) + { + return AssetJob.Missing(id); + } using var input = fileSystem.OpenRead(inputPath); var length = checked((int)input.Length); @@ -31,7 +35,12 @@ public static async Task> Decode(AssetId id, var asset = ReadPayload(ref reader, id, settings, decoder); var dependencies = ReadDependencies(ref reader); - return new Asset(id, asset, dependencies); + return AssetJob.Success(id, new Asset(id, asset, settings, dependencies)); + } + catch (Exception ex) + { + var edi = ExceptionDispatchInfo.Capture(ex); + return AssetJob.Failure(id, edi); } finally { diff --git a/source/CapriKit.AssetPipeline/AssetFileCache.cs b/source/CapriKit.AssetPipeline/AssetFileCache.cs new file mode 100644 index 0000000..761e00d --- /dev/null +++ b/source/CapriKit.AssetPipeline/AssetFileCache.cs @@ -0,0 +1,58 @@ +using CapriKit.IO; +using System.Buffers; + +namespace CapriKit.AssetPipeline; + +internal sealed class AssetFileCache(IVirtualFileSystem FileSystem, TranscoderCollection Transcoders) +{ + public async Task> Load(AssetId id, IAssetSettings settings) + where TAsset : class + { + var transcoder = Transcoders.Get(); + + var job = await AssetDecoder.Decode(id, transcoder, FileSystem); + return job.Match( + (_, asset) => + { + if (IsUpToDate(asset) && SettingsEqual(transcoder, asset.Settings, settings)) + { + return job; + } + return AssetJob.Missing(id); + }, + (_, _) => job, + (_) => job + ); + } + + private bool IsUpToDate(Asset asset) + where T : class + { + foreach (var (file, version) in asset.Dependencies) + { + if (!FileSystem.Exists(file)) + { + return false; + } + + var lastWrite = FileSystem.LastWriteTime(file); + if (version < lastWrite) + { + return false; + } + } + + return true; + } + + private static bool SettingsEqual(IAssetTranscoder transcoder, IAssetSettings embedded, IAssetSettings requested) + { + var embeddedWriter = new ArrayBufferWriter(); + transcoder.WriteSettings(embedded, embeddedWriter); + + var requestedWriter = new ArrayBufferWriter(); + transcoder.WriteSettings(requested, requestedWriter); + + return embeddedWriter.WrittenSpan.SequenceEqual(requestedWriter.WrittenSpan); + } +} diff --git a/source/CapriKit.AssetPipeline/AssetId.cs b/source/CapriKit.AssetPipeline/AssetId.cs deleted file mode 100644 index c98b808..0000000 --- a/source/CapriKit.AssetPipeline/AssetId.cs +++ /dev/null @@ -1,10 +0,0 @@ -using CapriKit.IO; - -namespace CapriKit.AssetPipeline; - -/// -/// Unique asset identifier -/// -/// Optional key to a sub-resources in Path. -/// Virtual file path that points to the file the asset originates from. -public record AssetId(string Key, FilePath Path); diff --git a/source/CapriKit.AssetPipeline/AssetManager.cs b/source/CapriKit.AssetPipeline/AssetManager.cs index f62ec91..4df68a6 100644 --- a/source/CapriKit.AssetPipeline/AssetManager.cs +++ b/source/CapriKit.AssetPipeline/AssetManager.cs @@ -1,7 +1,6 @@ using CapriKit.AssetPipeline.HotReloading; using CapriKit.IO; using Microsoft.Extensions.Logging; -using static CapriKit.AssetPipeline.AssetUtilities; namespace CapriKit.AssetPipeline; @@ -11,8 +10,10 @@ namespace CapriKit.AssetPipeline; public sealed class AssetManager { private readonly IVirtualFileSystem FileSystem; - private readonly Dictionary Transcoders = []; - private readonly AssetCache Cache; + private readonly TranscoderCollection Transcoders; + + private readonly AssetMemoryCache Cache; + private readonly AssetFileCache FileCache; private readonly HotSwapManager HotSwapManager; public AssetManager(ILoggerFactory logger, DirectoryPath rootDirectory) @@ -21,24 +22,27 @@ public AssetManager(ILoggerFactory logger, DirectoryPath rootDirectory) public AssetManager(ILoggerFactory logger, IVirtualFileSystem fileSystem) { FileSystem = fileSystem; - Cache = new AssetCache(); + Transcoders = new TranscoderCollection(); + Cache = new AssetMemoryCache(); + FileCache = new AssetFileCache(fileSystem, Transcoders); HotSwapManager = new HotSwapManager(logger, this, fileSystem); } - /// + /// public void PushScope() => Cache.PushScope(); - /// + /// public void PopScope() => Cache.PopScope(); + /// public void RegisterTranscoder(IAssetTranscoder transcoder) { - Transcoders[typeof(TAsset)] = transcoder; + Transcoders.Register(transcoder); } public Task Encode(AssetId id, IAssetSettings settings) { - return AssetEncoder.Encode(id, settings, GetTranscoder(), FileSystem); + return AssetEncoder.Encode(id, settings, Transcoders.Get(), FileSystem); } public Task Encode(AssetId id) @@ -49,83 +53,61 @@ public Task Encode(AssetId id) /// /// Immediately decodes an asset, bypasses cache and hot reloading mechanisms. - /// - public async Task Decode(AssetId id) - where TAsset : class - { - var asset = await DecodeInternal(id); - return asset.Value; - } - - internal Task> DecodeInternal(AssetId id) + /// + public Task> Decode(AssetId id) where TAsset : class { - return AssetDecoder.Decode(id, GetTranscoder(), FileSystem); + return AssetDecoder.Decode(id, Transcoders.Get(), FileSystem); } /// /// Loads an asset from the cache, decoding it from disk and building it first if it is missing or - /// out of date. Loaded assets are owned by the current scope, see . + /// out of date. The first time an asset is loaded it is put in the current scope, see . /// public async Task Load(AssetId id, IAssetSettings settings) where TAsset : class { - // If an asset was already loaded successfully we do not have to do an out-of-date check. - // Keeping live assets up-to-date is handled by the hot-reloading machinery. + // Live asset: ready for use. if (Cache.TryGet(id, out var entry)) { return entry; } - var asset = await DecodeOrBuild(id, settings); - Cache.Add(id, asset.Value); - return asset.Value; - } - - public Task Load(AssetId id) - where TAsset : class - { - return Load(id, default(NoSettings)); - } + // Matching encoded version available on disk: decode, register for hot swapping and return. + var getFromFileCache = await FileCache.Load(id, settings); + if (getFromFileCache.OnSuccess(out var cachedAsset)) + { + Cache.Add(id, cachedAsset.Value); + HotSwapManager.Track(cachedAsset, settings); + return cachedAsset.Value; + } - private async Task> DecodeOrBuild(AssetId id, IAssetSettings settings) - where TAsset : class - { - var transcoder = GetTranscoder(); - try + if (getFromFileCache.OnFailure(out var exception)) { - var asset = await AssetDecoder.Decode(id, transcoder, FileSystem); - if (IsUpToDate(asset, FileSystem)) - { - return asset; - } + // TODO: log + } - (asset.Value as IDisposable)?.Dispose(); + // Encoded version unavailable, out of date or created using different settings: + // Encode, decode, register for hot swapping and return. + await Encode(id, settings); + var getFromFullBuild = await Decode(id); + if (getFromFullBuild.OnSuccess(out var freshAsset)) + { + Cache.Add(id, freshAsset.Value); + HotSwapManager.Track(freshAsset, settings); } - catch (Exception ex) when (ex is FileNotFoundException or InvalidDataException) + + if (getFromFileCache.OnFailure(out var rebuildFailure)) { - // The asset was never built, or was built by a different transcoder version + rebuildFailure.Throw(); } - // Deliberately not guarded: if what we just built still fails to decode that is a bug in the - // transcoder and the exception should reach the caller - await Encode(id, settings); - return await AssetDecoder.Decode(id, transcoder, FileSystem); + throw new Exception($"Asset {id} could not be found"); } internal void HotSwap(TAsset instance, TAsset replacement) { - var transcoder = GetTranscoder(); + var transcoder = Transcoders.Get(); transcoder.HotSwap(instance, replacement); } - - private IAssetTranscoder GetTranscoder() - { - if (!Transcoders.TryGetValue(typeof(TAsset), out var transcoder)) - { - throw new InvalidOperationException($"No transcoder registered for asset type {typeof(TAsset).Name}"); - } - - return (IAssetTranscoder)transcoder; - } } diff --git a/source/CapriKit.AssetPipeline/AssetCache.cs b/source/CapriKit.AssetPipeline/AssetMemoryCache.cs similarity index 92% rename from source/CapriKit.AssetPipeline/AssetCache.cs rename to source/CapriKit.AssetPipeline/AssetMemoryCache.cs index f6c4202..e60b826 100644 --- a/source/CapriKit.AssetPipeline/AssetCache.cs +++ b/source/CapriKit.AssetPipeline/AssetMemoryCache.cs @@ -6,14 +6,14 @@ namespace CapriKit.AssetPipeline; /// Caches assets in stacked scopes that can be discarded in one go. /// Assets are released when the scope they were added in is popped or this class is disposed /// -public sealed class AssetCache : IDisposable +public sealed class AssetMemoryCache : IDisposable { private record CacheItem(int Scope, object Item, IDisposable? Disposable); private readonly Dictionary Cache; private int scope; - public AssetCache() + public AssetMemoryCache() { scope = 0; Cache = []; diff --git a/source/CapriKit.AssetPipeline/AssetUtilities.cs b/source/CapriKit.AssetPipeline/AssetUtilities.cs index c0fe1fe..1060446 100644 --- a/source/CapriKit.AssetPipeline/AssetUtilities.cs +++ b/source/CapriKit.AssetPipeline/AssetUtilities.cs @@ -22,24 +22,4 @@ public static void ThrowOnFileNotFound(FilePath path, IVirtualFileSystem fileSys throw new FileNotFoundException(null, path); } } - - public static bool IsUpToDate(Asset asset, IReadOnlyVirtualFileSystem fileSystem) - where T : class - { - foreach (var (file, version) in asset.Dependencies) - { - if (!fileSystem.Exists(file)) - { - return false; - } - - var lastWrite = fileSystem.LastWriteTime(file); - if (version < lastWrite) - { - return false; - } - } - - return true; - } } diff --git a/source/CapriKit.AssetPipeline/Dependency.cs b/source/CapriKit.AssetPipeline/Dependency.cs deleted file mode 100644 index bf04b1d..0000000 --- a/source/CapriKit.AssetPipeline/Dependency.cs +++ /dev/null @@ -1,5 +0,0 @@ -using CapriKit.IO; - -namespace CapriKit.AssetPipeline; - -public sealed record Dependency(FilePath File, DateTime Version); diff --git a/source/CapriKit.AssetPipeline/HotReloading/HotSwapManager.cs b/source/CapriKit.AssetPipeline/HotReloading/HotSwapManager.cs index 80bbd24..9260c04 100644 --- a/source/CapriKit.AssetPipeline/HotReloading/HotSwapManager.cs +++ b/source/CapriKit.AssetPipeline/HotReloading/HotSwapManager.cs @@ -106,8 +106,15 @@ private void HotSwapCompleted() { while (PendingHotSwaps.TryDequeue(out var result)) { - result.HotSwap(AssetManager, this); - LogReloadCompleted(Logger, result.Id); + try + { + result.HotSwap(AssetManager, this); + LogReloadCompleted(Logger, result.Id); + } + catch (Exception ex) + { + LogReloadFailed(Logger, result.Id, ex); + } } } diff --git a/source/CapriKit.AssetPipeline/HotReloading/HotSwappable.cs b/source/CapriKit.AssetPipeline/HotReloading/HotSwappable.cs index 0db06b0..310a60a 100644 --- a/source/CapriKit.AssetPipeline/HotReloading/HotSwappable.cs +++ b/source/CapriKit.AssetPipeline/HotReloading/HotSwappable.cs @@ -9,16 +9,26 @@ internal abstract class HotSwappable(AssetId id) public abstract void HotSwap(AssetManager manager, HotSwapManager hotSwapManager); } -internal sealed class HotSwappable(TAsset instance, Asset newParts, IAssetSettings settings) +internal sealed class HotSwappable(TAsset instance, AssetJob newParts, IAssetSettings settings) : HotSwappable(newParts.Id) where TAsset : class { public override void HotSwap(AssetManager assetManager, HotSwapManager hotSwapManager) { - assetManager.HotSwap(instance, newParts.Value); + if (newParts.OnSuccess(out var asset)) + { + assetManager.HotSwap(instance, asset.Value); - // Instance keeps being the active object, so keep tracking instance, but with the new dependencies - var toTrack = new Asset(newParts.Id, instance, newParts.Dependencies); - hotSwapManager.Track(toTrack, settings); + // Instance keeps being the active object, so keep tracking instance, but with the new dependencies + var toTrack = new Asset(asset.Id, instance, settings, asset.Dependencies); + hotSwapManager.Track(toTrack, settings); + } + + if (newParts.OnFailure(out var ex)) + { + ex.Throw(); + } + + throw new Exception($"Asset {Id} tracked for hot swapping could no longer be found"); } } diff --git a/source/CapriKit.AssetPipeline/HotReloading/Reloadable.cs b/source/CapriKit.AssetPipeline/HotReloading/Reloadable.cs index 199b89e..4d96099 100644 --- a/source/CapriKit.AssetPipeline/HotReloading/Reloadable.cs +++ b/source/CapriKit.AssetPipeline/HotReloading/Reloadable.cs @@ -40,7 +40,7 @@ public override async Task Reload(AssetManager manager, ConcurrentQueue(Id); + var hot = await manager.Decode(Id); queue.Enqueue(new HotSwappable(cold, hot, Settings)); } finally diff --git a/source/CapriKit.AssetPipeline/TranscoderCollection.cs b/source/CapriKit.AssetPipeline/TranscoderCollection.cs new file mode 100644 index 0000000..5ed865d --- /dev/null +++ b/source/CapriKit.AssetPipeline/TranscoderCollection.cs @@ -0,0 +1,21 @@ +namespace CapriKit.AssetPipeline; + +internal sealed class TranscoderCollection +{ + private readonly Dictionary Transcoders = []; + + public void Register(IAssetTranscoder transcoder) + { + Transcoders[typeof(TAsset)] = transcoder; + } + + public IAssetTranscoder Get() + { + if (!Transcoders.TryGetValue(typeof(TAsset), out var transcoder)) + { + throw new InvalidOperationException($"No transcoder registered for asset type {typeof(TAsset).Name}"); + } + + return (IAssetTranscoder)transcoder; + } +} diff --git a/source/CapriKit.Tests/AssetPipeline/AssetDecoderTests.cs b/source/CapriKit.Tests/AssetPipeline/AssetDecoderTests.cs index 907b100..a690336 100644 --- a/source/CapriKit.Tests/AssetPipeline/AssetDecoderTests.cs +++ b/source/CapriKit.Tests/AssetPipeline/AssetDecoderTests.cs @@ -15,14 +15,18 @@ public async Task Decode() var id = new AssetId("Main", "hello.txt"); await AssetEncoder.Encode(id, new NoSettings(), transcoder, fileSystem); - var envelope = await AssetDecoder.Decode(id, transcoder, fileSystem); + var job = await AssetDecoder.Decode(id, transcoder, fileSystem); FilePath expectedDependency = "hello.txt"; DateTime expectedTimeStamp = DateTime.Now; - await Assert.That(envelope.Value).IsEqualTo("HÉLLO"); - await Assert.That(envelope.Dependencies.Count).IsEqualTo(1); - await Assert.That(envelope.Dependencies.First().File).IsEqualTo(expectedDependency); - await Assert.That(envelope.Dependencies.First().Version) + + var success = job.OnSuccess(out var asset); + await Assert.That(success).IsTrue(); + + await Assert.That(asset!.Value).IsEqualTo("HÉLLO"); + await Assert.That(asset.Dependencies.Count).IsEqualTo(1); + await Assert.That(asset.Dependencies.First().File).IsEqualTo(expectedDependency); + await Assert.That(asset.Dependencies.First().Version) .IsBetween(expectedTimeStamp.AddMinutes(-1), expectedTimeStamp.AddMinutes(1)); } } From 1c9d897f86d6fffc8e5eede6114bc993d9d00277 Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Sun, 26 Jul 2026 22:20:29 +0200 Subject: [PATCH 21/53] Add review, fix later --- Research/AssetPipelineReview.md | 292 ++++++++++++++++++++++++++++++++ 1 file changed, 292 insertions(+) create mode 100644 Research/AssetPipelineReview.md diff --git a/Research/AssetPipelineReview.md b/Research/AssetPipelineReview.md new file mode 100644 index 0000000..650120c --- /dev/null +++ b/Research/AssetPipelineReview.md @@ -0,0 +1,292 @@ +# Asset Pipeline Review + +> Multi-agent review of `source/CapriKit.AssetPipeline` (branch `feature/asset_pipeline`), +> triggered by a worry that the caching, hot-reloading and `AssetJob` handling were +> too complex and mixed too many paradigms. Three review agents (correctness, +> clarity/paradigm-consistency, simplification) assessed the source without running any +> tools; every finding below was cross-checked against a manual read of the code. + +**Headline:** the worry was right for a concrete reason. The non-exhaustive `AssetJob` +consumption API (`OnSuccess`/`OnFailure`/`OnMissing`) is not just noisy — it *directly +caused* three correctness bugs by letting success paths fall through to `throw`. The +paradigm problem and the correctness problem are the same problem. + +## Overview + +| # | Cluster | Finding | Severity | Location | +|---|---------|---------|----------|----------| +| F1 | A · Result API | Successful full build never returns — falls through to `throw` | Critical | `AssetManager.cs:94-105` | +| F2 | A · Result API | Rebuild failure check inspects the **wrong** job variable | High | `AssetManager.cs:100` | +| F3 | A · Result API | Successful hot-swap falls through to `throw` (logged as failure) | High | `HotSwappable.cs:18-32` | +| F4 | C · Lifetime | `PopScope` mutates the dictionary while enumerating it | High | `AssetMemoryCache.cs:67-74` | +| F5 | B · Errors | `OpenRead` + `checked` cast throw *outside* the catch, escaping the `Failure` contract | Low | `AssetDecoder.cs:24-25` | +| F6 | A · Result API | `AssetJob` offers 4 consumption modes; the `On*` trio is non-exhaustive | High | `Asset.cs:48-94` | +| F7 | B · Errors | Five error paradigms; an exception is captured -> ferried -> re-thrown | High | decoder -> manager | +| F8 | B · Errors | The `Failure`/EDI state is probably unnecessary | Medium | `Asset.cs`, `AssetManager.cs` | +| F9 | A · Simplify | `AssetFileCache.Load`'s `Match` is an identity on 2 of 3 arms | Medium | `AssetFileCache.cs:14-25` | +| F10 | A · Simplify | `SettingsEqual` re-serializes both sides on every load | Medium | `AssetFileCache.cs:48-57` | +| F11 | A · Simplify | The "success->register / else rethrow / else throw" tail is duplicated | High | `AssetManager.cs` + `HotSwappable.cs` | +| F12 | C · Lifetime | Two ownership models (scope-stack vs `WeakReference`); pop doesn't untrack | High | `AssetMemoryCache` + `HotSwapManager` | +| F13 | C · Threading | `Track` mutates plain dictionaries off the main thread | Medium | `HotSwapManager.cs:49-53` | +| F14 | D · Naming | "Job", `On*`, and dual "Cache"/"Load" each name two things | Medium | cross-cutting | +| F15 | D · API surface | Public API leaks `Encode`/`Decode`/`AssetJob` internals | Medium | `AssetManager`, `Asset.cs` | +| F16 | D · API surface | `IAssetTranscoder` mixes public/internal members across two arities | Low | `IAssetTranscoder.cs` | + +The four clusters map onto the three original worries: **A + B = the `AssetJob` +handling**, **C = caching + hot reloading**, **D = the cross-cutting naming/surface tax**. +Fix A first — it contains the only shippable blockers. + +--- + +## Cluster A — The result API, and the bugs it caused + +`AssetJob` lets you consume a three-state value with a **non-exhaustive** idiom +(`if (job.OnSuccess(out …)) { … }`), which silently collapses the other two states into a +fall-through. Nothing forces you to handle all three, so a missing `return` compiles +cleanly and ships. + +### F1 — Successful build never returns *(Critical)* +`AssetManager.cs:94` + +```csharp +var getFromFullBuild = await Decode(id); +if (getFromFullBuild.OnSuccess(out var freshAsset)) +{ + Cache.Add(id, freshAsset.Value); + HotSwapManager.Track(freshAsset, settings); + // no return — control falls through … +} +// … +throw new Exception($"Asset {id} could not be found"); // reached on success +``` + +**Failure scenario:** memory miss + disk miss + a *successful* build -> the asset is +built, cached and tracked, then `Load` throws "could not be found". (A second call would +hit the memory cache and succeed, making the bug maddening to diagnose.) Every first-time +load of a not-yet-built asset throws. + +**Fix:** add `return freshAsset.Value;` inside the success block. + +### F2 — Wrong job variable in the failure check *(High)* +`AssetManager.cs:100` + +```csharp +if (getFromFileCache.OnFailure(out var rebuildFailure)) { rebuildFailure.Throw(); } +``` + +After the rebuild, this inspects the **original file-cache** job, not `getFromFullBuild`. +So a genuine *build* error is discarded (you get the generic "could not be found"), while a +stale *cache* error can be re-thrown even though the rebuild is what actually ran. +**Fix:** check `getFromFullBuild.OnFailure(...)`. This also proves F8 — the "remember the +cache error and rethrow it later" behaviour isn't actually relied upon. + +### F3 — Successful hot-swap throws *(High)* +`HotSwappable.cs:18` + +Same shape: the `OnSuccess` block does the swap and re-tracks, but doesn't `return`, so it +falls to `throw new Exception($"Asset {Id} … could no longer be found")` on line 32. That +exception is caught by `HotSwapManager.HotSwapCompleted` and logged via `LogReloadFailed` — +so **every successful hot reload is reported as a failure**, after its side effects already +ran. **Fix:** add `return;` after the success block. + +### F6 — Four ways to consume one value; two are unsafe *(High)* +`Asset.cs` + +`AssetJob` exposes `OnSuccess`/`OnFailure`/`OnMissing` (imperative, **non-exhaustive**) +*and* two `Match` overloads (functional, **exhaustive**). Offering both means every reader +must learn two APIs and every author must pick the safe one unaided — and F1-F3 are what +happens when they don't. Secondary smell: `OnSuccess(out Asset? asset)` returns a +**nullable** even on success, so a caller who trusts the out over the bool invents yet +another path. + +**Recommendation — keep the exhaustive one.** The whole point of the tri-state is to *not* +lose the Failure/Missing distinction, and only `Match` (or a `switch` on an explicit state +enum) enforces that. + +- Delete the `On*` trio. Consume via `Match`, or add + `enum AssetJobState { Success, Failure, Missing }` + a non-nullable payload accessor and + `switch` on it — a `switch` expression still gets exhaustiveness warnings; three + independent `bool`s never can. +- Make the success payload non-nullable. + +### F11 — Unify the duplicated tail *(High)* +The "on success add/track/use; else rethrow EDI; else throw not-found" shape is hand-rolled +in **both** `AssetManager.Load` and `HotSwappable.HotSwap` — which is exactly why the same +fall-through slipped into both. Extract it once so it can't recur: + +```csharp +private TAsset Register(Asset asset, IAssetSettings settings) where T : class +{ + Cache.Add(asset.Id, asset.Value); + HotSwapManager.Track(asset, settings); + return asset.Value; +} + +[DoesNotReturn] +private static TAsset Unavailable(AssetJob job, AssetId id) where T : class +{ + if (job.OnFailure(out var edi)) { edi.Throw(); } + throw new AssetNotFoundException(id); +} +``` + +The `Load` tail then becomes +`return getFromFullBuild.OnSuccess(out var fresh) ? Register(fresh, settings) : Unavailable(getFromFullBuild, id);` +— F1 and F2 both become structurally impossible. + +### F9 — Dead `Match` ceremony *(Medium)* +`AssetFileCache.Load` uses a 3-arm `Match` where the failure and missing arms both just +`return job` — only the success arm has logic. Collapse to early returns: + +```csharp +var job = await AssetDecoder.Decode(id, transcoder, FileSystem); +if (job.OnSuccess(out var asset)) + return IsUpToDate(asset) && SettingsEqual(transcoder, asset.Settings, settings) + ? job : AssetJob.Missing(id); +return job; // failure and missing pass straight through +``` + +### F10 — `SettingsEqual` serializes twice per load *(Medium)* +It writes both the embedded and requested settings into fresh `ArrayBufferWriter`s +just to compare bytes — two allocations + two serializations on every disk hit. If settings +are `record`/`record struct`, compare structurally instead: + +```csharp +// default method on IAssetTranscoder +bool SettingsEqual(TSettings a, TSettings b) => EqualityComparer.Default.Equals(a, b); +``` + +Cheaper fallback if you keep bytes: `AssetDecoder` already read the settings bytes off disk +— stash them on the `Asset` and serialize only the *requested* side once. + +--- + +## Cluster B — Too many error paradigms + +### F7 — Five vocabularies, and a full round-trip *(High)* +The module speaks (1) plain `throw`, (2) `ExceptionDispatchInfo` capture-and-rethrow, +(3) the tri-state `AssetJob`, (4) `bool`-try (`TryGet`, the `On*` trio), and (5) nullable +`out`. They don't layer — they convert into each other in a circle: `AssetDecoder` catches +**every** exception and demotes it to `Failure(EDI)`; that failure rides through +`AssetFileCache`, is held across an `await` while `AssetManager` re-encodes, and is finally +re-thrown at the bottom of `Load`. Tracing "what happens on a corrupt file?" means holding +the demoted error in your head across the whole method. + +**Fix:** make `AssetJob` the *only* currency inside the pipeline and convert to an exception +exactly once, at the single public `Load` seam — by `Match`-ing the **final** job (not +re-reading a stale one). Keep `ExceptionDispatchInfo` **only** where it earns its keep: +preserving a stack trace across the `await`/thread-pool hop in hot reload. Everywhere the +exception never crosses a thread, store a plain `Exception`. + +### F8 — The `Failure` state may be dead weight *(Medium)* +`Failure` exists solely to carry an EDI so `Load` can *defer* a rethrow — but F2 shows that +deferral is buggy and unrelied-upon, and `Missing` vs `Failure` are otherwise treated +identically (both fall through to rebuild). Consider collapsing to **two states + +exceptions**: `Decode` returns `Asset?` (`null` = missing) and simply *throws* on a +corrupt file; `Load` wraps the disk read in `try/catch`, logs, and rebuilds. That deletes +the `Failure` state, the EDI field, `OnFailure`, and all the remember-then-rethrow +bookkeeping. This is the single biggest reduction in paradigm count — decide first whether +you ever genuinely need "surface the *cache* error only when the *rebuild* also fails" (the +current code suggests not). + +### F5 — Decoder error-path gap *(Low)* +`AssetDecoder.cs:24-25`: `fileSystem.OpenRead(...)` and `checked((int)input.Length)` run +*before* the `try` (line 28), so an IO/sharing error or a >2 GB `OverflowException` escapes +as a raw throw instead of the `Failure` job the method otherwise promises. **Fix:** move +both inside the `try` (or accept the inconsistency once F8 makes throwing the norm). + +--- + +## Cluster C — Lifetime & threading + +### F4 — `PopScope` crashes and leaks *(High)* +`AssetMemoryCache.cs:67` + +```csharp +foreach (var (key, value) in Cache) { if (value.Scope >= scope) { value.Disposable?.Dispose(); Cache.Remove(key); } } +``` + +`Dictionary` bumps its version on `Remove`, so the next `MoveNext` throws +`InvalidOperationException: Collection was modified` — on essentially every real scope pop +that held an asset. The loop then aborts, so the remaining assets in that scope are never +disposed (leak). **Fix:** snapshot the keys first: + +```csharp +foreach (var key in Cache.Where(kv => kv.Value.Scope >= scope).Select(kv => kv.Key).ToList()) +{ + Cache[key].Disposable?.Dispose(); + Cache.Remove(key); +} +``` + +### F12 — Two ownership models that can disagree *(High)* +`AssetMemoryCache` says the **cache owns** assets (holds `IDisposable?`, disposes on pop). +`Reloadable` says they're **not owned** (holds only a `WeakReference`, treats +"collected" as normal). Both can't be the authority on "is this alive?" Because the cache +strongly holds the disposable for the whole scope, the weak-ref can't die *until* pop — and +crucially **nothing untracks the `Reloadable` from `HotSwapManager` on `PopScope`**. So +after a pop, `Tracked`/`Dependents` still reference the now-disposed asset; a file change +can hot-swap a **disposed instance** (this is the `// TODO: cold can be alive but disposed`). + +**Fix:** make **scope the single owner**. Have `Reloadable`/hot-reload hold the *AssetId* +(a cache key) instead of a `WeakReference`, and look the live instance up in the +cache at reload time — reloads then naturally stop when the scope is popped. Add an +`Untrack(id)` called from `PopScope`. Reserve `WeakReference` only if you deliberately +support caller-owned assets the cache does *not* dispose (and document that split). + +### F13 — `Track` races the main thread *(Medium)* +`HotSwapManager.Track` writes plain `Dictionary`s (`Tracked`, `Dependents`). It's documented +main-thread, but `AssetManager.Load` calls it *after* `await`s (lines 78, 94); with no +game-loop `SynchronizationContext` those continuations run on thread-pool threads, while +`ProcessUpdates` reads/enumerates the same dictionaries each frame on the main thread -> +data race / mid-enumeration throw. **Fix:** marshal `Track` onto the main thread (queue it +like hot-swaps and apply in `ProcessUpdates`), or guard both dictionaries with a lock. +(Severity depends on the threading model — if `Load` is always awaited on the main thread +with a context installed, this drops to Low.) + +--- + +## Cluster D — Naming & public surface + +### F14 — Names that each mean two things *(Medium)* +- **`AssetJob`** is not a job — it's an already-computed result, but "Job" implies something + you `Start`/`await`. Rename -> `AssetResult` / `AssetOutcome`. +- **`On*`** is C#'s convention for *event handlers* (`OnClick`), yet here they're predicates + — and the same words are the `Match` callback parameters. Rename survivors to the + `Try*`/`Is*` convention. +- **"Cache"** names both the volatile in-memory store and the on-disk store; **"Load"** + names both the full orchestration (`AssetManager.Load`) and decode-one-file + (`AssetFileCache.Load`). Rename by role: `AssetStore` / `AssetDiskCache`, and give the + file method a narrower verb (`TryReadFromDisk`). + +### F15 — Public surface leaks the plumbing *(Medium)* +A user needs: register transcoders, `Load(id)`, scope for lifetime. But `Encode`, +`Decode` (returning `AssetJob`), and thereby the whole tri-state/EDI machinery are public +too. **Fix:** make `Encode`/`Decode`/`AssetJob` `internal` (with `InternalsVisibleTo` for +tests). Every paradigm hidden from users is a finding that stops being their problem. This +also lets F9's `AssetFileCache` fold into a private `AssetManager.TryReadFromDisk` — after +the trims above it's ~5 lines and holds nothing the manager can't reach. + +### F16 — `IAssetTranscoder` density *(Low)* +The two-arity settings-erasure is a legitimate, clever technique — but it mixes +accessibilities *inside* one interface (public `HotSwap` beside internal `Encode`) and +bridges via default-interface methods, so "how is a transcoder invoked?" needs two +interfaces read at once. **Fix:** move the erased members to an internal +`IAssetTranscoderCore`, and add one XML-doc line on it: "you implement +`IAssetTranscoder`; the erased base is bridged for you." + +--- + +## Suggested order of attack + +1. **F1, F2, F3, F4** — four small, mechanical fixes; correctness blockers (add two + `return`s, fix one variable, snapshot before removing). +2. **F11 + F6** — unify the tail and move to exhaustive consumption, so F1-F3 can't return. +3. **F8 + F7** — decide whether `Failure`/EDI stays; collapsing to nullable+throw erases + most of the "too many paradigms" feeling. +4. **F12, F13** — lifetime authority + `Track` threading (the hot-reload half of the worry). +5. **F14, F15, F9, F10, F16** — naming, surface, and the smaller trims. + +The three agents were unanimous that the tri-state result API is the load-bearing issue: it +caused the correctness bugs *and* it's the main source of the "multiple paradigms" feeling — +fix that cluster and most of the rest gets smaller on its own. From 7c22e64cf9badbeca5e7c4642d0b8da93c0ec051 Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Sat, 8 Aug 2026 23:24:16 +0200 Subject: [PATCH 22/53] WIP: a different look at the asset pipeline --- .../CapriKit.AssetPipeline/v2/AssetCache.cs | 68 +++++++++++ .../CapriKit.AssetPipeline/v2/AssetManager.cs | 113 ++++++++++++++++++ 2 files changed, 181 insertions(+) create mode 100644 source/CapriKit.AssetPipeline/v2/AssetCache.cs create mode 100644 source/CapriKit.AssetPipeline/v2/AssetManager.cs diff --git a/source/CapriKit.AssetPipeline/v2/AssetCache.cs b/source/CapriKit.AssetPipeline/v2/AssetCache.cs new file mode 100644 index 0000000..590cd66 --- /dev/null +++ b/source/CapriKit.AssetPipeline/v2/AssetCache.cs @@ -0,0 +1,68 @@ +using System.Diagnostics.CodeAnalysis; + +namespace CapriKit.AssetPipeline.v2; + +internal sealed class AssetCache +{ + private class CacheEntry(object Asset, int RefCount) + { + public readonly object asset = Asset; + public int refCount = RefCount; + } + + private readonly Lock Lock = new(); + + private readonly Dictionary Entries = []; + + public void Put(AssetId id, TAsset asset) + where TAsset : class + { + lock (Lock) + { + if (Entries.ContainsKey(id)) + { + throw new Exception($"Cache already contains asset: {id}."); + } + + var entry = new CacheEntry(asset, 1); + Entries.Add(id, entry); + } + } + + public bool TryLease(AssetId id, [NotNullWhen(true)] out TAsset? asset) + where TAsset : class + { + lock (Lock) + { + if (Entries.TryGetValue(id, out var entry)) + { + entry.refCount = entry.refCount + 1; + asset = (TAsset)entry.asset; + return true; + } + } + + asset = default; + return false; + } + + public bool TryRelease(AssetId id, [NotNullWhen(true)] out IDisposable? disposable) + where TAsset : class + { + lock (Lock) + { + if (Entries.TryGetValue(id, out var entry)) + { + entry.refCount = entry.refCount - 1; + if (entry.refCount == 0 && entry.asset is IDisposable disposableAsset) + { + disposable = disposableAsset; + return true; + } + } + } + + disposable = default; + return false; + } +} diff --git a/source/CapriKit.AssetPipeline/v2/AssetManager.cs b/source/CapriKit.AssetPipeline/v2/AssetManager.cs new file mode 100644 index 0000000..79cefa7 --- /dev/null +++ b/source/CapriKit.AssetPipeline/v2/AssetManager.cs @@ -0,0 +1,113 @@ +using CapriKit.IO; +using System.Diagnostics.CodeAnalysis; + +namespace CapriKit.AssetPipeline.v2; + +public record class BuildMetaData(AssetId Id, IReadOnlyList Dependencies); + +public sealed class AssetManager +{ + private readonly IVirtualFileSystem FileSystem; + private readonly TranscoderCollection Transcoders; + + /// + public void RegisterTranscoder(IAssetTranscoder transcoder) + { + Transcoders.Register(transcoder); + } + + public async Task Load(AssetId id, IAssetSettings settings) + where TAsset : class + { + if (LoadFromCache(id, out var cachedAsset)) + { + return cachedAsset; + } + + var transcoder = Transcoders.Get(); + if (LoadBuildMetadata(id, settings, transcoder, out var build) && IsUpToDate(build)) + { + return await Decode(id, settings, transcoder); + } + + // The asset was not build or is out of date + + if (!SourceFileExists(id)) + { + throw new FileNotFoundException("Could not find primary file to build asset from", id.Path); + } + + await Encode(id, settings, transcoder); + var asset = await Decode(id, settings, transcoder); + RegisterAsset(id, asset, settings); + + return asset; + } + + public void Unload(AssetId id) + { + // TODO: Decrease the reference count in the cache and if refcount == 0 remove from hot reloading and dispose + throw new NotImplementedException(); + } + + /// Must be called from the main thread. + public void Update() + { + // TODO: Perform work that can only be done on the main thread (like hot-reloading) + throw new NotImplementedException(); + } + + private async Task Encode(AssetId id, IAssetSettings settings, IAssetTranscoder transcoder) where TAsset : class + { + throw new NotImplementedException(); + } + + private async Task Decode(AssetId id, IAssetSettings settings, IAssetTranscoder transcoder) where TAsset : class + { + // LOad from file + // Add to cache + throw new NotImplementedException(); + } + + private bool LoadFromCache(AssetId id, [NotNullWhen(true)] out TAsset? asset) + where TAsset : class + { + throw new NotImplementedException(); + } + + private bool LoadBuildMetadata(AssetId id, IAssetSettings settings, IAssetTranscoder transcoder, [NotNullWhen(true)] out BuildMetaData? build) + where TAsset : class + { + throw new NotImplementedException(); + } + + private bool IsUpToDate(BuildMetaData build) + { + foreach (var (file, version) in build.Dependencies) + { + if (!FileSystem.Exists(file)) + { + return false; + } + + var lastWrite = FileSystem.LastWriteTime(file); + if (version < lastWrite) + { + return false; + } + } + + return true; + } + private bool SourceFileExists(AssetId id) + { + throw new NotImplementedException(); + } + + private void RegisterAsset(AssetId id, TAsset asset, IAssetSettings settings) where TAsset : class + { + // Add to cache + // Register for hot reloading + throw new NotImplementedException(); + } +} From e0bacaaa8408486d03b060da14f400a537af294a Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Sun, 9 Aug 2026 14:59:11 +0200 Subject: [PATCH 23/53] WIP: a different look at the asset pipeline #2 --- source/CapriKit.AssetPipeline/v2/Asset.cs | 17 +++ .../CapriKit.AssetPipeline/v2/AssetCache.cs | 56 +++++--- .../CapriKit.AssetPipeline/v2/AssetDecoder.cs | 136 ++++++++++++++++++ .../CapriKit.AssetPipeline/v2/AssetEncoder.cs | 69 +++++++++ .../CapriKit.AssetPipeline/v2/AssetManager.cs | 101 ++++++------- .../v2/AssetUtilities.cs | 25 ++++ .../CapriKit.AssetPipeline/v2/HotReloader.cs | 54 +++++++ .../v2/IAssetTranscoder.cs | 37 +++++ 8 files changed, 426 insertions(+), 69 deletions(-) create mode 100644 source/CapriKit.AssetPipeline/v2/Asset.cs create mode 100644 source/CapriKit.AssetPipeline/v2/AssetDecoder.cs create mode 100644 source/CapriKit.AssetPipeline/v2/AssetEncoder.cs create mode 100644 source/CapriKit.AssetPipeline/v2/AssetUtilities.cs create mode 100644 source/CapriKit.AssetPipeline/v2/HotReloader.cs create mode 100644 source/CapriKit.AssetPipeline/v2/IAssetTranscoder.cs diff --git a/source/CapriKit.AssetPipeline/v2/Asset.cs b/source/CapriKit.AssetPipeline/v2/Asset.cs new file mode 100644 index 0000000..c9f35a2 --- /dev/null +++ b/source/CapriKit.AssetPipeline/v2/Asset.cs @@ -0,0 +1,17 @@ +using CapriKit.IO; + +namespace CapriKit.AssetPipeline.v2; + +/// +/// Unique asset identifier +/// +/// Optional key to a sub-resources in Path. +/// Virtual file path that points to the file the asset originates from. +public record AssetId(string Key, FilePath Path); + +public record Asset(AssetId Id, TAsset Value, AssetBuildMetaData BuildMetaData) + where TAsset : class; + +public record class AssetBuildMetaData(Guid TranscoderId, int TranscoderVersion, TSettings Settings, IReadOnlyList Dependencies); + +public sealed record Dependency(FilePath File, DateTime Version); diff --git a/source/CapriKit.AssetPipeline/v2/AssetCache.cs b/source/CapriKit.AssetPipeline/v2/AssetCache.cs index 590cd66..d0de3fc 100644 --- a/source/CapriKit.AssetPipeline/v2/AssetCache.cs +++ b/source/CapriKit.AssetPipeline/v2/AssetCache.cs @@ -2,30 +2,29 @@ namespace CapriKit.AssetPipeline.v2; -internal sealed class AssetCache +internal sealed class AssetCache : IDisposable { - private class CacheEntry(object Asset, int RefCount) + private class Line(object Asset, int RefCount) { public readonly object asset = Asset; public int refCount = RefCount; } private readonly Lock Lock = new(); - - private readonly Dictionary Entries = []; + private readonly Dictionary Lines = []; public void Put(AssetId id, TAsset asset) where TAsset : class { lock (Lock) { - if (Entries.ContainsKey(id)) + if (Lines.ContainsKey(id)) { throw new Exception($"Cache already contains asset: {id}."); } - var entry = new CacheEntry(asset, 1); - Entries.Add(id, entry); + var entry = new Line(asset, 1); + Lines.Add(id, entry); } } @@ -34,7 +33,7 @@ public bool TryLease(AssetId id, [NotNullWhen(true)] out TAsset? asset) { lock (Lock) { - if (Entries.TryGetValue(id, out var entry)) + if (Lines.TryGetValue(id, out var entry)) { entry.refCount = entry.refCount + 1; asset = (TAsset)entry.asset; @@ -46,23 +45,42 @@ public bool TryLease(AssetId id, [NotNullWhen(true)] out TAsset? asset) return false; } - public bool TryRelease(AssetId id, [NotNullWhen(true)] out IDisposable? disposable) - where TAsset : class + public void Return(AssetId id) { lock (Lock) { - if (Entries.TryGetValue(id, out var entry)) + var entry = Lines[id]; + entry.refCount = entry.refCount - 1; + } + } + + public void Collect() + { + List? toCollect = null; + foreach (var (key, value) in Lines) + { + if (value.refCount <= 0) { - entry.refCount = entry.refCount - 1; - if (entry.refCount == 0 && entry.asset is IDisposable disposableAsset) - { - disposable = disposableAsset; - return true; - } + toCollect = toCollect ?? []; + toCollect.Add(key); } } - disposable = default; - return false; + if (toCollect == null) { return; } + + foreach (var key in toCollect) + { + Lines.Remove(key, out var entry); + (entry as IDisposable)?.Dispose(); + } + } + + public void Dispose() + { + foreach (var value in Lines.Values) + { + (value as IDisposable)?.Dispose(); + } + Lines.Clear(); } } diff --git a/source/CapriKit.AssetPipeline/v2/AssetDecoder.cs b/source/CapriKit.AssetPipeline/v2/AssetDecoder.cs new file mode 100644 index 0000000..d059968 --- /dev/null +++ b/source/CapriKit.AssetPipeline/v2/AssetDecoder.cs @@ -0,0 +1,136 @@ +using CapriKit.IO; +using CapriKit.IO.Streams; +using System.Buffers; +using static CapriKit.AssetPipeline.v2.AssetUtilities; + +namespace CapriKit.AssetPipeline.v2; + +/// +/// Decodes the generic asset envelope and, using a specialized IAssetTranscoder, the asset itself +/// +internal static class AssetDecoder +{ + public static async Task> Decode(AssetId id, IAssetTranscoder decoder, IReadOnlyVirtualFileSystem fileSystem) + where TAsset : class + { + var inputPath = ToEncodedFilePath(id); + if (!fileSystem.Exists(inputPath)) + { + throw new FileNotFoundException($"Could not find file: {inputPath} to load asset: {id}", id.Path); + } + + using var input = fileSystem.OpenRead(inputPath); + var length = checked((int)input.Length); + var buffer = ArrayPool.Shared.Rent(length); + + try + { + await input.ReadExactlyAsync(buffer.AsMemory(0, length)); + var reader = SequenceReaders.Create(buffer, 0, length); + + var (encoderId, encoderVersion) = ReadHeader(ref reader); + ThrowOnDecoderMismatch(id, inputPath, encoderId, encoderVersion, decoder); + + var settings = ReadSettings(ref reader, decoder); + var asset = ReadPayload(ref reader, id, decoder, settings); + var dependencies = ReadDependencies(ref reader); + + var buildMetaData = new AssetBuildMetaData(encoderId, encoderVersion, settings, dependencies); + return new Asset(id, asset, buildMetaData); + } + finally + { + ArrayPool.Shared.Return(buffer); + } + } + + public static async Task?> TryDecodeBuildMetaData(AssetId id, IAssetTranscoder decoder, IReadOnlyVirtualFileSystem fileSystem) + where TAsset : class + { + try + { + var inputPath = ToEncodedFilePath(id); + if (!fileSystem.Exists(inputPath)) + { + throw new FileNotFoundException($"Could not find file: {inputPath} to load asset: {id}", id.Path); + } + + using var input = fileSystem.OpenRead(inputPath); + var length = checked((int)input.Length); + var buffer = ArrayPool.Shared.Rent(length); + + try + { + await input.ReadExactlyAsync(buffer.AsMemory(0, length)); + var reader = SequenceReaders.Create(buffer, 0, length); + + var (encoderId, encoderVersion) = ReadHeader(ref reader); + var settings = ReadSettings(ref reader, decoder); + SkipPayload(ref reader); + var dependencies = ReadDependencies(ref reader); + return new AssetBuildMetaData(encoderId, encoderVersion, settings, dependencies); + } + finally + { + ArrayPool.Shared.Return(buffer); + } + } + catch + { + return null; + } + } + + private static (Guid id, int version) ReadHeader(ref SequenceReader reader) + { + var id = reader.ReadGuid(); + var version = reader.ReadInt32(); + return (id, version); + } + + private static TSettings ReadSettings(ref SequenceReader reader, IAssetTranscoder decoder) + where TAsset : class + { + var settingsLength = reader.ReadInt32(); + var settingsReader = reader.SliceUnread(settingsLength); + return decoder.ReadSettings(ref settingsReader); + } + + private static TAsset ReadPayload(ref SequenceReader reader, AssetId id, IAssetTranscoder decoder, TSettings settings) + where TAsset : class + { + var payloadLength = reader.ReadInt32(); + var payloadReader = reader.SliceUnread(payloadLength); + return decoder.Decode(id, settings, ref payloadReader); + } + + private static void SkipPayload(ref SequenceReader reader) + { + var payloadLength = reader.ReadInt32(); + reader.Advance(payloadLength); + } + + private static List ReadDependencies(ref SequenceReader reader) + { + var count = reader.ReadInt32(); + var dependencies = new List(count); + for (var i = 0; i < count; i++) + { + var lastWriteTicks = reader.ReadInt64(); + var lastWrite = new DateTime(lastWriteTicks); + var filePathString = reader.ReadString(); + var filePath = new FilePath(filePathString); + dependencies.Add(new Dependency(filePath, lastWrite)); + } + return dependencies; + } + + private static void ThrowOnDecoderMismatch(AssetId id, FilePath file, Guid fileId, int fileVersion, IAssetTranscoder decoder) + { + if (fileId != decoder.Id || fileVersion != decoder.Version) + { + throw new InvalidDataException( + $"Decoder mismatch. Asset: {id} found in file: {file}, was encoded using {fileId}:v{fileVersion} but the current decoder is {decoder.Id}:v{decoder.Version}."); + } + } +} diff --git a/source/CapriKit.AssetPipeline/v2/AssetEncoder.cs b/source/CapriKit.AssetPipeline/v2/AssetEncoder.cs new file mode 100644 index 0000000..fd48902 --- /dev/null +++ b/source/CapriKit.AssetPipeline/v2/AssetEncoder.cs @@ -0,0 +1,69 @@ +using CapriKit.IO; +using CapriKit.IO.Streams; +using System.Buffers; +using System.IO.Pipelines; +using static CapriKit.AssetPipeline.v2.AssetUtilities; + +namespace CapriKit.AssetPipeline.v2; + +// File format: [encoder id][encoder version][settings length][settings][payload length][payload][dependency count][dependencies] + +/// +/// Encodes the generic asset envelope and, using a specialized IAssetTranscoder, the asset itself +/// +internal static class AssetEncoder +{ + public static async Task Encode(AssetId id, IAssetTranscoder encoder, TSettings settings, IVirtualFileSystem fileSystem) + where TAsset : class + { + ThrowOnFileNotFound(id.Path, fileSystem); + var outputPath = ToEncodedFilePath(id); + + using var output = fileSystem.CreateReadWrite(outputPath); + var writer = PipeWriter.Create(output); + var spy = fileSystem.SpyOn(); + + WriteHeader(writer, encoder); + WriteSettings(writer, encoder, settings); + await WritePayload(writer, id, encoder, settings, spy); + WriteDependencies(writer, spy); + + await writer.FlushAsync(); + await writer.CompleteAsync(); + } + + private static void WriteHeader(PipeWriter writer, IAssetTranscoder encoder) + { + writer.Write(encoder.Id); + writer.Write(encoder.Version); + } + + private static void WriteSettings(PipeWriter writer, IAssetTranscoder encoder, TSettings settings) + where TAsset : class + { + var buffer = new ArrayBufferWriter(); + encoder.WriteSettings(settings, buffer); + writer.Write(buffer.WrittenCount); + writer.Write(buffer.WrittenSpan); + } + + private static async Task WritePayload(PipeWriter writer, AssetId id, IAssetTranscoder encoder, TSettings settings, VirtualFileSystemSpy spy) + where TAsset : class + { + var payload = new ArrayBufferWriter(); + await encoder.Encode(id, settings, spy, payload); + writer.Write(payload.WrittenCount); + writer.Write(payload.WrittenSpan); + } + + private static void WriteDependencies(PipeWriter writer, VirtualFileSystemSpy spy) + { + writer.Write(spy.OpenedFiles.Count); + foreach (var dependency in spy.OpenedFiles) + { + var lastWrite = spy.LastWriteTime(dependency); + writer.Write(lastWrite.Ticks); + writer.Write(dependency); + } + } +} diff --git a/source/CapriKit.AssetPipeline/v2/AssetManager.cs b/source/CapriKit.AssetPipeline/v2/AssetManager.cs index 79cefa7..1a09fc6 100644 --- a/source/CapriKit.AssetPipeline/v2/AssetManager.cs +++ b/source/CapriKit.AssetPipeline/v2/AssetManager.cs @@ -1,88 +1,88 @@ using CapriKit.IO; -using System.Diagnostics.CodeAnalysis; +using Microsoft.Extensions.Logging; +using System.Buffers; namespace CapriKit.AssetPipeline.v2; -public record class BuildMetaData(AssetId Id, IReadOnlyList Dependencies); - -public sealed class AssetManager +// TODO: Add logging +public sealed class AssetManager : IDisposable { private readonly IVirtualFileSystem FileSystem; - private readonly TranscoderCollection Transcoders; + private readonly AssetCache Cache; + private readonly ILogger Logger; - /// - public void RegisterTranscoder(IAssetTranscoder transcoder) + public AssetManager(ILoggerFactory logger, IVirtualFileSystem fileSystem) { - Transcoders.Register(transcoder); + Logger = logger.CreateLogger(); + FileSystem = fileSystem; + Cache = new AssetCache(); } - public async Task Load(AssetId id, IAssetSettings settings) + public async Task Load(AssetId id, IAssetTranscoder transcoder, TSettings settings) where TAsset : class { - if (LoadFromCache(id, out var cachedAsset)) + // Check if the asset was loaded before + if (Cache.TryLease(id, out var cachedAsset)) { return cachedAsset; } - var transcoder = Transcoders.Get(); - if (LoadBuildMetadata(id, settings, transcoder, out var build) && IsUpToDate(build)) + // If not, check if it can be loaded from an up-to-date build + var build = await AssetDecoder.TryDecodeBuildMetaData(id, transcoder, FileSystem); + if (build != null && IsUpToDate(transcoder, settings, build)) { - return await Decode(id, settings, transcoder); + var upToDateAsset = await AssetDecoder.Decode(id, transcoder, FileSystem); + RegisterAsset(upToDateAsset, transcoder); + return upToDateAsset.Value; } - // The asset was not build or is out of date - - if (!SourceFileExists(id)) + // If not, try to rebuild and load the asset + if (!FileSystem.Exists(id.Path)) { throw new FileNotFoundException("Could not find primary file to build asset from", id.Path); } - await Encode(id, settings, transcoder); - var asset = await Decode(id, settings, transcoder); - RegisterAsset(id, asset, settings); + await AssetEncoder.Encode(id, transcoder, settings, FileSystem); + var freshAsset = await AssetDecoder.Decode(id, transcoder, FileSystem); + RegisterAsset(freshAsset, transcoder); - return asset; + return freshAsset.Value; } public void Unload(AssetId id) { - // TODO: Decrease the reference count in the cache and if refcount == 0 remove from hot reloading and dispose - throw new NotImplementedException(); + Cache.Return(id); } /// Must be called from the main thread. public void Update() { // TODO: Perform work that can only be done on the main thread (like hot-reloading) - throw new NotImplementedException(); + Cache.Collect(); } - private async Task Encode(AssetId id, IAssetSettings settings, IAssetTranscoder transcoder) where TAsset : class + private bool IsUpToDate(IAssetTranscoder transcoder, TSettings settings, AssetBuildMetaData build) + where TAsset : class { - throw new NotImplementedException(); - } + // Transcoders differ + if (transcoder.Id != build.TranscoderId || transcoder.Version != build.TranscoderVersion) + { + return false; + } - private async Task Decode(AssetId id, IAssetSettings settings, IAssetTranscoder transcoder) where TAsset : class - { - // LOad from file - // Add to cache - throw new NotImplementedException(); - } + // Settings differ + var inUse = new ArrayBufferWriter(); + transcoder.WriteSettings(settings, inUse); - private bool LoadFromCache(AssetId id, [NotNullWhen(true)] out TAsset? asset) - where TAsset : class - { - throw new NotImplementedException(); - } + var inFile = new ArrayBufferWriter(); + transcoder.WriteSettings(build.Settings, inFile); - private bool LoadBuildMetadata(AssetId id, IAssetSettings settings, IAssetTranscoder transcoder, [NotNullWhen(true)] out BuildMetaData? build) - where TAsset : class - { - throw new NotImplementedException(); - } + if (!inUse.WrittenSpan.SequenceEqual(inFile.WrittenSpan)) + { + return false; + } - private bool IsUpToDate(BuildMetaData build) - { + // Dependencies have changed foreach (var (file, version) in build.Dependencies) { if (!FileSystem.Exists(file)) @@ -99,15 +99,16 @@ private bool IsUpToDate(BuildMetaData build) return true; } - private bool SourceFileExists(AssetId id) + + private void RegisterAsset(Asset asset, IAssetTranscoder transcoder) + where TAsset : class { - throw new NotImplementedException(); + // TODO: Register for hot reloading + Cache.Put(asset.Id, asset.Value); } - private void RegisterAsset(AssetId id, TAsset asset, IAssetSettings settings) where TAsset : class + public void Dispose() { - // Add to cache - // Register for hot reloading - throw new NotImplementedException(); + Cache.Dispose(); } } diff --git a/source/CapriKit.AssetPipeline/v2/AssetUtilities.cs b/source/CapriKit.AssetPipeline/v2/AssetUtilities.cs new file mode 100644 index 0000000..f894d16 --- /dev/null +++ b/source/CapriKit.AssetPipeline/v2/AssetUtilities.cs @@ -0,0 +1,25 @@ +using CapriKit.IO; + +namespace CapriKit.AssetPipeline.v2; + +internal static class AssetUtilities +{ + public static FilePath ToEncodedFilePath(AssetId id) + { + if (string.IsNullOrEmpty(id.Key)) + { + return $"{id.Path}.cka"; + } + + var key = IOUtilities.EscapeFileName(id.Key); + return $"{id.Path}.{key}.cka"; + } + + public static void ThrowOnFileNotFound(FilePath path, IVirtualFileSystem fileSystem) + { + if (!fileSystem.Exists(path)) + { + throw new FileNotFoundException(null, path); + } + } +} diff --git a/source/CapriKit.AssetPipeline/v2/HotReloader.cs b/source/CapriKit.AssetPipeline/v2/HotReloader.cs new file mode 100644 index 0000000..b4e572c --- /dev/null +++ b/source/CapriKit.AssetPipeline/v2/HotReloader.cs @@ -0,0 +1,54 @@ +using CapriKit.IO; +using Microsoft.Extensions.Logging; +using System.Collections.Concurrent; + +namespace CapriKit.AssetPipeline.v2; + +internal sealed class HotReloader : IDisposable +{ + private abstract class Reloadable(AssetId id) + { + public AssetId Id { get; } = id; + public abstract Task Reload(IVirtualFileSystem fileSystem, ConcurrentQueue hotSwapActionQueue); + } + + private sealed class Reloadable : Reloadable + where TAsset : class + { + private readonly WeakReference> Instance; + private readonly IAssetTranscoder Transcoder; + + + public Reloadable(AssetId id, Asset asset, IAssetTranscoder transcoder) + : base(id) + { + Instance = new WeakReference>(asset); + Transcoder = transcoder; + } + + public override async Task Reload(IVirtualFileSystem fileSystem, ConcurrentQueue hotSwapActionQueue) + { + if (!Instance.TryGetTarget(out var cold)) { return; } + + await AssetEncoder.Encode(cold.Id, Transcoder, cold.BuildMetaData.Settings, fileSystem); + var hot = await AssetDecoder.Decode(cold.Id, Transcoder, fileSystem); + hotSwapActionQueue.Enqueue(() => Transcoder.HotSwap(cold.Value, hot.Value)); + } + } + + + + private static readonly TimeSpan MinWaitTime = TimeSpan.FromSeconds(0.5); + private readonly ILogger Logger; + + private readonly AssetManager AssetManager; + private readonly IReadOnlyVirtualFileSystem FileSystem; + + private long lastFileChange; + + public void Track(Asset asset, IAssetTranscoder transcoder) + where TAsset : class + { + + } +} diff --git a/source/CapriKit.AssetPipeline/v2/IAssetTranscoder.cs b/source/CapriKit.AssetPipeline/v2/IAssetTranscoder.cs new file mode 100644 index 0000000..14e5f32 --- /dev/null +++ b/source/CapriKit.AssetPipeline/v2/IAssetTranscoder.cs @@ -0,0 +1,37 @@ +using CapriKit.IO; +using System.Buffers; + +namespace CapriKit.AssetPipeline.v2; + + +public interface IAssetTranscoder +{ + Guid Id { get; } + int Version { get; } +} + +// TODO: improve documentation +public interface IAssetTranscoder : IAssetTranscoder + where TAsset : class +{ + // Asynchronous since we expect the encoder to read external files + public Task Encode(AssetId id, TSettings settings, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer); + + // Synchronous by design: the envelope owns all file IO and hands the decoder an + // in-memory payload. The reader's buffer is only valid for the duration of the call, + // decoders must copy out anything they want to keep. + public TAsset Decode(AssetId id, TSettings settings, ref SequenceReader reader); + + public void WriteSettings(TSettings settings, IBufferWriter writer); + + public TSettings ReadSettings(ref SequenceReader reader); + + /// + /// Moves the contents of into . + /// keeps its identity and stays the live object that callers + /// already hold references to. The transcoder is responsible for cleaning-up any + /// orphaned resources. After calling this method must no longer + /// be used or referenced. + /// + void HotSwap(TAsset instance, TAsset newParts); +} From aa3af474475631b1f733c74994c9902911d96e4c Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Sun, 9 Aug 2026 15:56:09 +0200 Subject: [PATCH 24/53] WIP: a different look at the asset pipeline #3 --- .../v2/HotReloadManager.cs | 169 ++++++++++++++++++ .../v2/HotReloadable.cs | 37 ++++ .../CapriKit.AssetPipeline/v2/HotReloader.cs | 54 ------ 3 files changed, 206 insertions(+), 54 deletions(-) create mode 100644 source/CapriKit.AssetPipeline/v2/HotReloadManager.cs create mode 100644 source/CapriKit.AssetPipeline/v2/HotReloadable.cs delete mode 100644 source/CapriKit.AssetPipeline/v2/HotReloader.cs diff --git a/source/CapriKit.AssetPipeline/v2/HotReloadManager.cs b/source/CapriKit.AssetPipeline/v2/HotReloadManager.cs new file mode 100644 index 0000000..9aeb4f0 --- /dev/null +++ b/source/CapriKit.AssetPipeline/v2/HotReloadManager.cs @@ -0,0 +1,169 @@ +using CapriKit.Concurrency.Async; +using CapriKit.IO; +using CapriKit.IO.Watchers; +using Microsoft.Extensions.Logging; +using System.Collections.Concurrent; +using System.Diagnostics; + +namespace CapriKit.AssetPipeline.v2; + +/// +/// Facilitates hot reloading and hot swapping of assets. Tracks the files used to create an asset +/// and triggers a rebuild on file changes. Takes care of threading and only performs the final +/// hot swap when is called. +/// +internal sealed partial class HotReloadManager : IDisposable +{ + private static readonly TimeSpan MinWaitTime = TimeSpan.FromSeconds(0.5); + private readonly ILogger Logger; + + private readonly IVirtualFileSystem FileSystem; + + private readonly Dictionary Tracked; + private readonly Dictionary> Dependents; + + private readonly IVirtualFileSystemWatcher Watcher; + private readonly FileSystemEventQueue FileChances; + private readonly HashSet PendingRebuilds; + private readonly ConcurrentQueue PendingReloads; + + private long lastFileChange; + private bool isReloading; + + public HotReloadManager(ILoggerFactory logger, IVirtualFileSystem fileSystem) + { + Logger = logger.CreateLogger(); + FileSystem = fileSystem; + + Tracked = []; + Dependents = []; + + Watcher = fileSystem.Watch(DirectoryPath.Empty); // Watch for all changed, usually fileSystem is a ScopedVirtualFileSystem + FileChances = new FileSystemEventQueue(Watcher); + PendingRebuilds = []; + PendingReloads = []; + + lastFileChange = Stopwatch.GetTimestamp(); + isReloading = false; + } + + public void Track(Asset asset, IAssetTranscoder transcoder) + where TAsset : class + { + Tracked[asset.Id] = new HotReloadable(asset, transcoder); + foreach (var dependency in asset.BuildMetaData.Dependencies) + { + var file = dependency.File; + if (Dependents.TryGetValue(file, out var ids)) + { + ids.Add(asset.Id); + } + else + { + ids = [asset.Id]; + Dependents.Add(file, ids); + } + } + } + + public void Update() + { + if (isReloading) + { + return; + } + + DrainFileChanges(); + var elapsed = Stopwatch.GetElapsedTime(lastFileChange); + if (elapsed > MinWaitTime) + { + ReloadOne(); + } + + HotSwapPending(); + } + + private void DrainFileChanges() + { + while (FileChances.TryDequeue(out var @event)) + { + if (Dependents.TryGetValue(@event.File, out var dependents)) + { + lastFileChange = Stopwatch.GetTimestamp(); + foreach (var id in dependents) + { + PendingRebuilds.Add(id); + LogPendingReload(Logger, @event.File, id); + } + } + } + } + private void ReloadOne() + { + if (PendingRebuilds.Count > 0) + { + var id = PendingRebuilds.First(); + PendingRebuilds.Remove(id); + + isReloading = true; + + LogReloadStarted(Logger, id); + + var reloadable = Tracked[id]; + reloadable.Reload(FileSystem, PendingReloads) + .FireAndForget(ex => LogReloadFailed(Logger, id, ex), () => + { + isReloading = false; + LogReloadCompleted(Logger, id); + }); + } + } + + private void HotSwapPending() + { + while (PendingReloads.TryDequeue(out var action)) + { + try + { + LogHotSwapStarted(Logger, action.Id); + action.PerformHotSwap(); + LogHotSwapCompleted(Logger, action.Id); + } + catch (Exception ex) + { + LogHotSwapFailed(Logger, action.Id, ex); + } + } + } + + public void Dispose() + { + Watcher.Stop(); + Tracked.Clear(); + Dependents.Clear(); + PendingRebuilds.Clear(); + PendingReloads.Clear(); + } + + [LoggerMessage(Level = LogLevel.Information, Message = "Detected file change: {path}, affecting asset: {asset}")] + private static partial void LogPendingReload(ILogger logger, FilePath path, AssetId asset); + + + [LoggerMessage(Level = LogLevel.Information, Message = "Reloading asset started: {asset}")] + private static partial void LogReloadStarted(ILogger logger, AssetId asset); + + [LoggerMessage(Level = LogLevel.Error, Message = "Reloading asset completed: {asset}")] + private static partial void LogReloadCompleted(ILogger logger, AssetId asset); + + [LoggerMessage(Level = LogLevel.Error, Message = "Reloading asset failed: {asset}")] + private static partial void LogReloadFailed(ILogger logger, AssetId asset, Exception exception); + + [LoggerMessage(Level = LogLevel.Information, Message = "Hot-swapping asset started: {asset}")] + private static partial void LogHotSwapStarted(ILogger logger, AssetId asset); + + [LoggerMessage(Level = LogLevel.Information, Message = "Hot-swapping asset completed: {asset}")] + private static partial void LogHotSwapCompleted(ILogger logger, AssetId asset); + + [LoggerMessage(Level = LogLevel.Error, Message = "Hot-swapping asset failed: {asset}")] + private static partial void LogHotSwapFailed(ILogger logger, AssetId asset, Exception exception); +} diff --git a/source/CapriKit.AssetPipeline/v2/HotReloadable.cs b/source/CapriKit.AssetPipeline/v2/HotReloadable.cs new file mode 100644 index 0000000..6f91b2f --- /dev/null +++ b/source/CapriKit.AssetPipeline/v2/HotReloadable.cs @@ -0,0 +1,37 @@ +using CapriKit.IO; +using System.Collections.Concurrent; + +namespace CapriKit.AssetPipeline.v2; + +internal sealed record HotSwapAction(AssetId Id, Action PerformHotSwap); + +internal abstract class HotReloadable(AssetId id) +{ + public AssetId Id { get; } = id; + public abstract Task Reload(IVirtualFileSystem fileSystem, ConcurrentQueue hotSwapActionQueue); +} + +internal sealed class HotReloadable : HotReloadable + where TAsset : class +{ + private readonly WeakReference> Instance; + private readonly IAssetTranscoder Transcoder; + + + public HotReloadable(Asset asset, IAssetTranscoder transcoder) + : base(asset.Id) + { + Instance = new WeakReference>(asset); + Transcoder = transcoder; + } + + public override async Task Reload(IVirtualFileSystem fileSystem, ConcurrentQueue hotSwapActionQueue) + { + if (!Instance.TryGetTarget(out var cold)) { return; } + + await AssetEncoder.Encode(cold.Id, Transcoder, cold.BuildMetaData.Settings, fileSystem); + var hot = await AssetDecoder.Decode(cold.Id, Transcoder, fileSystem); + + hotSwapActionQueue.Enqueue(new HotSwapAction(cold.Id, () => Transcoder.HotSwap(cold.Value, hot.Value))); + } +} diff --git a/source/CapriKit.AssetPipeline/v2/HotReloader.cs b/source/CapriKit.AssetPipeline/v2/HotReloader.cs deleted file mode 100644 index b4e572c..0000000 --- a/source/CapriKit.AssetPipeline/v2/HotReloader.cs +++ /dev/null @@ -1,54 +0,0 @@ -using CapriKit.IO; -using Microsoft.Extensions.Logging; -using System.Collections.Concurrent; - -namespace CapriKit.AssetPipeline.v2; - -internal sealed class HotReloader : IDisposable -{ - private abstract class Reloadable(AssetId id) - { - public AssetId Id { get; } = id; - public abstract Task Reload(IVirtualFileSystem fileSystem, ConcurrentQueue hotSwapActionQueue); - } - - private sealed class Reloadable : Reloadable - where TAsset : class - { - private readonly WeakReference> Instance; - private readonly IAssetTranscoder Transcoder; - - - public Reloadable(AssetId id, Asset asset, IAssetTranscoder transcoder) - : base(id) - { - Instance = new WeakReference>(asset); - Transcoder = transcoder; - } - - public override async Task Reload(IVirtualFileSystem fileSystem, ConcurrentQueue hotSwapActionQueue) - { - if (!Instance.TryGetTarget(out var cold)) { return; } - - await AssetEncoder.Encode(cold.Id, Transcoder, cold.BuildMetaData.Settings, fileSystem); - var hot = await AssetDecoder.Decode(cold.Id, Transcoder, fileSystem); - hotSwapActionQueue.Enqueue(() => Transcoder.HotSwap(cold.Value, hot.Value)); - } - } - - - - private static readonly TimeSpan MinWaitTime = TimeSpan.FromSeconds(0.5); - private readonly ILogger Logger; - - private readonly AssetManager AssetManager; - private readonly IReadOnlyVirtualFileSystem FileSystem; - - private long lastFileChange; - - public void Track(Asset asset, IAssetTranscoder transcoder) - where TAsset : class - { - - } -} From ea9db91fd47cd24fbd2fd9a8f292f0b87a373ec8 Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Sun, 9 Aug 2026 16:02:13 +0200 Subject: [PATCH 25/53] WIP: a different look at the asset pipeline #4 --- .../v2/NoSettingsTranscoder.cs | 29 +++++++++++++++++++ 1 file changed, 29 insertions(+) create mode 100644 source/CapriKit.AssetPipeline/v2/NoSettingsTranscoder.cs diff --git a/source/CapriKit.AssetPipeline/v2/NoSettingsTranscoder.cs b/source/CapriKit.AssetPipeline/v2/NoSettingsTranscoder.cs new file mode 100644 index 0000000..718b9c5 --- /dev/null +++ b/source/CapriKit.AssetPipeline/v2/NoSettingsTranscoder.cs @@ -0,0 +1,29 @@ +using CapriKit.IO; +using System.Buffers; + +namespace CapriKit.AssetPipeline.v2; + +internal readonly struct NoSettings; + +public abstract class NoSettingsTranscoder(Guid id, int version) : IAssetTranscoder + where TAsset : class +{ + public Guid Id { get; } = id; + public int Version { get; } = version; + + public abstract Task Encode(AssetId id, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer); + + public abstract TAsset Decode(AssetId id, ref SequenceReader reader); + + public abstract void HotSwap(TAsset instance, TAsset newParts); + + TAsset IAssetTranscoder.Decode(AssetId id, NoSettings settings, ref SequenceReader reader) + => Decode(id, ref reader); + + Task IAssetTranscoder.Encode(AssetId id, NoSettings settings, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer) + => Encode(id, fileSystem, writer); + + NoSettings IAssetTranscoder.ReadSettings(ref SequenceReader reader) => default; + + void IAssetTranscoder.WriteSettings(NoSettings settings, IBufferWriter writer) { } +} From 5a4a056a9fe548091077a100baac3f68cba86c2d Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Sun, 9 Aug 2026 16:34:44 +0200 Subject: [PATCH 26/53] Complete redo asset pipeline --- .../AssetManagerExtensions.cs | 14 ++ .../Shaders/VertexShaderTranscoder.cs | 4 +- source/CapriKit.AssetPipeline/Asset.cs | 85 +--------- .../{v2 => }/AssetCache.cs | 7 +- source/CapriKit.AssetPipeline/AssetDecoder.cs | 89 +++++++--- source/CapriKit.AssetPipeline/AssetEncoder.cs | 13 +- .../CapriKit.AssetPipeline/AssetFileCache.cs | 58 ------- source/CapriKit.AssetPipeline/AssetManager.cs | 157 ++++++++++-------- .../AssetMemoryCache.cs | 88 ---------- .../{v2 => }/HotReloadManager.cs | 18 +- .../{v2 => }/HotReloadable.cs | 15 +- .../HotReloading/HotSwapManager.cs | 137 --------------- .../HotReloading/HotSwappable.cs | 34 ---- .../HotReloading/Reloadable.cs | 51 ------ .../IAssetTranscoder.cs | 87 ++++------ .../NoSettingsTranscoder.cs | 30 ++-- .../ServiceCollectionExtensions.cs | 3 +- .../TranscoderCollection.cs | 21 --- source/CapriKit.AssetPipeline/v2/Asset.cs | 17 -- .../CapriKit.AssetPipeline/v2/AssetDecoder.cs | 136 --------------- .../CapriKit.AssetPipeline/v2/AssetEncoder.cs | 69 -------- .../CapriKit.AssetPipeline/v2/AssetManager.cs | 114 ------------- .../v2/AssetUtilities.cs | 25 --- .../v2/IAssetTranscoder.cs | 37 ----- .../v2/NoSettingsTranscoder.cs | 29 ---- .../AssetPipeline/AssetDecoderTests.cs | 32 ---- .../AssetPipeline/AssetEncoderTests.cs | 44 ----- .../AssetPipeline/AssetManagerTests.cs | 41 ----- .../AssetPipeline/DummyTranscoder.cs | 43 ----- .../AssetPipeline/RepeatTranscoder.cs | 49 ------ 30 files changed, 261 insertions(+), 1286 deletions(-) create mode 100644 source/CapriKit.AssetPipeline.DirectX11/AssetManagerExtensions.cs rename source/CapriKit.AssetPipeline/{v2 => }/AssetCache.cs (77%) delete mode 100644 source/CapriKit.AssetPipeline/AssetFileCache.cs delete mode 100644 source/CapriKit.AssetPipeline/AssetMemoryCache.cs rename source/CapriKit.AssetPipeline/{v2 => }/HotReloadManager.cs (93%) rename source/CapriKit.AssetPipeline/{v2 => }/HotReloadable.cs (62%) delete mode 100644 source/CapriKit.AssetPipeline/HotReloading/HotSwapManager.cs delete mode 100644 source/CapriKit.AssetPipeline/HotReloading/HotSwappable.cs delete mode 100644 source/CapriKit.AssetPipeline/HotReloading/Reloadable.cs delete mode 100644 source/CapriKit.AssetPipeline/TranscoderCollection.cs delete mode 100644 source/CapriKit.AssetPipeline/v2/Asset.cs delete mode 100644 source/CapriKit.AssetPipeline/v2/AssetDecoder.cs delete mode 100644 source/CapriKit.AssetPipeline/v2/AssetEncoder.cs delete mode 100644 source/CapriKit.AssetPipeline/v2/AssetManager.cs delete mode 100644 source/CapriKit.AssetPipeline/v2/AssetUtilities.cs delete mode 100644 source/CapriKit.AssetPipeline/v2/IAssetTranscoder.cs delete mode 100644 source/CapriKit.AssetPipeline/v2/NoSettingsTranscoder.cs delete mode 100644 source/CapriKit.Tests/AssetPipeline/AssetDecoderTests.cs delete mode 100644 source/CapriKit.Tests/AssetPipeline/AssetEncoderTests.cs delete mode 100644 source/CapriKit.Tests/AssetPipeline/AssetManagerTests.cs delete mode 100644 source/CapriKit.Tests/AssetPipeline/DummyTranscoder.cs delete mode 100644 source/CapriKit.Tests/AssetPipeline/RepeatTranscoder.cs diff --git a/source/CapriKit.AssetPipeline.DirectX11/AssetManagerExtensions.cs b/source/CapriKit.AssetPipeline.DirectX11/AssetManagerExtensions.cs new file mode 100644 index 0000000..42dbe28 --- /dev/null +++ b/source/CapriKit.AssetPipeline.DirectX11/AssetManagerExtensions.cs @@ -0,0 +1,14 @@ +using CapriKit.AssetPipeline.DirectX11.Shaders; +using CapriKit.DirectX11; +using CapriKit.DirectX11.Resources.Shaders; + +namespace CapriKit.AssetPipeline.DirectX11; + +public static class AssetManagerExtensions +{ + public static Task LoadVertexShader(this AssetManager assetManager, Device device, AssetId id) + { + var transcoder = new VertexShaderTranscoder(device); + return assetManager.Load(id, transcoder, default); + } +} diff --git a/source/CapriKit.AssetPipeline.DirectX11/Shaders/VertexShaderTranscoder.cs b/source/CapriKit.AssetPipeline.DirectX11/Shaders/VertexShaderTranscoder.cs index dc3f23a..f1824d0 100644 --- a/source/CapriKit.AssetPipeline.DirectX11/Shaders/VertexShaderTranscoder.cs +++ b/source/CapriKit.AssetPipeline.DirectX11/Shaders/VertexShaderTranscoder.cs @@ -8,7 +8,7 @@ namespace CapriKit.AssetPipeline.DirectX11.Shaders; public sealed class VertexShaderTranscoder(Device device) : NoSettingsTranscoder(Guid.Parse("{CA3CB37D-9880-4B61-AB09-EBC17E7533E6}"), 1) { - public async override Task Encode(AssetId id, NoSettings _, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer) + public override async Task Encode(AssetId id, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer) { var source = await fileSystem.ReadAllText(id.Path); var includePath = id.Path.Directory; @@ -16,7 +16,7 @@ public async override Task Encode(AssetId id, NoSettings _, IRead ShaderTranscoder.WriteCommon(bytes.Common, writer); } - public override IVertexShader Decode(AssetId id, NoSettings _, ref SequenceReader reader) + public override IVertexShader Decode(AssetId id, ref SequenceReader reader) { var common = ShaderTranscoder.ReadCommon(ref reader); return ShaderCompiler.CreateVertexShader(new VertexShaderByteCode(common), device); diff --git a/source/CapriKit.AssetPipeline/Asset.cs b/source/CapriKit.AssetPipeline/Asset.cs index 45f5c6f..bf4033d 100644 --- a/source/CapriKit.AssetPipeline/Asset.cs +++ b/source/CapriKit.AssetPipeline/Asset.cs @@ -1,6 +1,4 @@ using CapriKit.IO; -using System.Diagnostics.CodeAnalysis; -using System.Runtime.ExceptionServices; namespace CapriKit.AssetPipeline; @@ -11,86 +9,9 @@ namespace CapriKit.AssetPipeline; /// Virtual file path that points to the file the asset originates from. public record AssetId(string Key, FilePath Path); - -/// -/// An active asset -/// -/// -/// The unique id, refers to a virtual file location (id.Path) and if required a sub-resource in that file (id.Key). -/// The asset -/// The settings used for encoding/decoding -/// Files that this asset depends on, if any of these files changed the asset needs te be rebuild. -public sealed record Asset(AssetId Id, TAsset Value, IAssetSettings Settings, IReadOnlyList Dependencies) +internal record Asset(AssetId Id, TAsset Value, AssetBuildMetaData BuildMetaData) where TAsset : class; -public sealed record Dependency(FilePath File, DateTime Version); - - -/// -/// Result of building an asset -/// -public sealed class AssetJob - where TAsset : class - -{ - private readonly Asset? Asset; - private readonly ExceptionDispatchInfo? Exception; - - private AssetJob(AssetId id, Asset? asset, ExceptionDispatchInfo? exception) - { - Id = id; - Asset = asset; - Exception = exception; - } - - public AssetId Id { get; } - - public bool OnSuccess([NotNullWhen(true)] out Asset? asset) - { - asset = Asset; - return asset != null; - } - - public bool OnFailure([NotNullWhen(true)] out ExceptionDispatchInfo? exception) - { - exception = Exception; - return exception != null; - } - - public bool OnMissing() - { - return Asset == null && Exception == null; - } - - public static AssetJob Failure(AssetId id, ExceptionDispatchInfo exception) - { - return new AssetJob(id, null, exception); - } - - public static AssetJob Success(AssetId id, Asset asset) - { - return new AssetJob(id, asset, null); - } - - public static AssetJob Missing(AssetId id) - { - return new AssetJob(id, null, null); - } - - public void Match(Action> onSuccess, Action onFailure, Action onMissing) - { - if (Asset != null) { onSuccess(Id, Asset); } - else if (Exception != null) { onFailure(Id, Exception); } - else { onMissing(Id); } - } - - public TReturn Match(Func, TReturn> onSuccess, - Func onFailure, - Func onMissing) - { - if (Asset != null) { return onSuccess(Id, Asset); } - if (Exception != null) { return onFailure(Id, Exception); } - return onMissing(Id); - } -} +internal record class AssetBuildMetaData(Guid TranscoderId, int TranscoderVersion, TSettings Settings, IReadOnlyList Dependencies); +internal sealed record Dependency(FilePath File, DateTime Version); diff --git a/source/CapriKit.AssetPipeline/v2/AssetCache.cs b/source/CapriKit.AssetPipeline/AssetCache.cs similarity index 77% rename from source/CapriKit.AssetPipeline/v2/AssetCache.cs rename to source/CapriKit.AssetPipeline/AssetCache.cs index d0de3fc..eff1e42 100644 --- a/source/CapriKit.AssetPipeline/v2/AssetCache.cs +++ b/source/CapriKit.AssetPipeline/AssetCache.cs @@ -1,7 +1,11 @@ using System.Diagnostics.CodeAnalysis; -namespace CapriKit.AssetPipeline.v2; +namespace CapriKit.AssetPipeline; +/// +/// Simple cache that uses reference counting to decide when to clean-up a resource. Assets can be leased and returned at any time (though the class requires single-threaded access). +/// The actual disposing of objects only happens when the main thread calls . +/// internal sealed class AssetCache : IDisposable { private class Line(object Asset, int RefCount) @@ -54,6 +58,7 @@ public void Return(AssetId id) } } + // TODO: can we dispose objects in the background or would that make the GPU unhappy when assets like textures are disposed at random times? public void Collect() { List? toCollect = null; diff --git a/source/CapriKit.AssetPipeline/AssetDecoder.cs b/source/CapriKit.AssetPipeline/AssetDecoder.cs index db428b2..4ad7369 100644 --- a/source/CapriKit.AssetPipeline/AssetDecoder.cs +++ b/source/CapriKit.AssetPipeline/AssetDecoder.cs @@ -1,7 +1,6 @@ using CapriKit.IO; using CapriKit.IO.Streams; using System.Buffers; -using System.Runtime.ExceptionServices; using static CapriKit.AssetPipeline.AssetUtilities; namespace CapriKit.AssetPipeline; @@ -11,36 +10,33 @@ namespace CapriKit.AssetPipeline; /// internal static class AssetDecoder { - public static async Task> Decode(AssetId id, - IAssetTranscoder decoder, IVirtualFileSystem fileSystem) + public static async Task> Decode(AssetId id, IAssetTranscoder decoder, IReadOnlyVirtualFileSystem fileSystem) where TAsset : class { var inputPath = ToEncodedFilePath(id); if (!fileSystem.Exists(inputPath)) { - return AssetJob.Missing(id); + throw new FileNotFoundException($"Could not find file: {inputPath} to load asset: {id}", id.Path); } using var input = fileSystem.OpenRead(inputPath); var length = checked((int)input.Length); - var buffer = ArrayPool.Shared.Rent(length); + try { await input.ReadExactlyAsync(buffer.AsMemory(0, length)); var reader = SequenceReaders.Create(buffer, 0, length); - ReadHeader(ref reader, decoder, inputPath); + var (encoderId, encoderVersion) = ReadHeader(ref reader); + ThrowOnDecoderMismatch(id, inputPath, encoderId, encoderVersion, decoder); + var settings = ReadSettings(ref reader, decoder); - var asset = ReadPayload(ref reader, id, settings, decoder); + var asset = ReadPayload(ref reader, id, decoder, settings); var dependencies = ReadDependencies(ref reader); - return AssetJob.Success(id, new Asset(id, asset, settings, dependencies)); - } - catch (Exception ex) - { - var edi = ExceptionDispatchInfo.Capture(ex); - return AssetJob.Failure(id, edi); + var buildMetaData = new AssetBuildMetaData(encoderId, encoderVersion, settings, dependencies); + return new Asset(id, asset, buildMetaData); } finally { @@ -48,34 +44,72 @@ public static async Task> Decode(AssetId id, } } - private static void ReadHeader(ref SequenceReader reader, IAssetTranscoder decoder, FilePath path) + public static async Task?> TryDecodeBuildMetaData(AssetId id, IAssetTranscoder decoder, IReadOnlyVirtualFileSystem fileSystem) + where TAsset : class { - var id = reader.ReadGuid(); - var version = reader.ReadInt32(); + try + { + var inputPath = ToEncodedFilePath(id); + if (!fileSystem.Exists(inputPath)) + { + throw new FileNotFoundException($"Could not find file: {inputPath} to load asset: {id}", id.Path); + } + + using var input = fileSystem.OpenRead(inputPath); + var length = checked((int)input.Length); + var buffer = ArrayPool.Shared.Rent(length); + + try + { + await input.ReadExactlyAsync(buffer.AsMemory(0, length)); + var reader = SequenceReaders.Create(buffer, 0, length); - if (id != decoder.Id || version != decoder.Version) + var (encoderId, encoderVersion) = ReadHeader(ref reader); + var settings = ReadSettings(ref reader, decoder); + SkipPayload(ref reader); + var dependencies = ReadDependencies(ref reader); + return new AssetBuildMetaData(encoderId, encoderVersion, settings, dependencies); + } + finally + { + ArrayPool.Shared.Return(buffer); + } + } + catch { - throw new InvalidDataException( - $"Cannot decode {path}, it was encoded by {id} v{version} but the decoder is {decoder.Id} v{decoder.Version}"); + return null; } } - private static IAssetSettings ReadSettings(ref SequenceReader reader, - IAssetTranscoder decoder) + private static (Guid id, int version) ReadHeader(ref SequenceReader reader) + { + var id = reader.ReadGuid(); + var version = reader.ReadInt32(); + return (id, version); + } + + private static TSettings ReadSettings(ref SequenceReader reader, IAssetTranscoder decoder) + where TAsset : class { var settingsLength = reader.ReadInt32(); var settingsReader = reader.SliceUnread(settingsLength); return decoder.ReadSettings(ref settingsReader); } - private static TAsset ReadPayload(ref SequenceReader reader, - AssetId id, IAssetSettings settings, IAssetTranscoder decoder) + private static TAsset ReadPayload(ref SequenceReader reader, AssetId id, IAssetTranscoder decoder, TSettings settings) + where TAsset : class { var payloadLength = reader.ReadInt32(); var payloadReader = reader.SliceUnread(payloadLength); return decoder.Decode(id, settings, ref payloadReader); } + private static void SkipPayload(ref SequenceReader reader) + { + var payloadLength = reader.ReadInt32(); + reader.Advance(payloadLength); + } + private static List ReadDependencies(ref SequenceReader reader) { var count = reader.ReadInt32(); @@ -90,4 +124,13 @@ private static List ReadDependencies(ref SequenceReader reader } return dependencies; } + + private static void ThrowOnDecoderMismatch(AssetId id, FilePath file, Guid fileId, int fileVersion, IAssetTranscoder decoder) + { + if (fileId != decoder.Id || fileVersion != decoder.Version) + { + throw new InvalidDataException( + $"Decoder mismatch. Asset: {id} found in file: {file}, was encoded using {fileId}:v{fileVersion} but the current decoder is {decoder.Id}:v{decoder.Version}."); + } + } } diff --git a/source/CapriKit.AssetPipeline/AssetEncoder.cs b/source/CapriKit.AssetPipeline/AssetEncoder.cs index 9434779..2b5542b 100644 --- a/source/CapriKit.AssetPipeline/AssetEncoder.cs +++ b/source/CapriKit.AssetPipeline/AssetEncoder.cs @@ -13,7 +13,8 @@ namespace CapriKit.AssetPipeline; /// internal static class AssetEncoder { - public static async Task Encode(AssetId id, IAssetSettings settings, IAssetTranscoder encoder, IVirtualFileSystem fileSystem) + public static async Task Encode(AssetId id, IAssetTranscoder encoder, TSettings settings, IVirtualFileSystem fileSystem) + where TAsset : class { ThrowOnFileNotFound(id.Path, fileSystem); var outputPath = ToEncodedFilePath(id); @@ -24,7 +25,7 @@ public static async Task Encode(AssetId id, IAssetSettings setti WriteHeader(writer, encoder); WriteSettings(writer, encoder, settings); - await WritePayload(writer, id, settings, encoder, spy); + await WritePayload(writer, id, encoder, settings, spy); WriteDependencies(writer, spy); await writer.FlushAsync(); @@ -37,15 +38,17 @@ private static void WriteHeader(PipeWriter writer, IAssetTranscoder encoder) writer.Write(encoder.Version); } - private static void WriteSettings(PipeWriter writer, IAssetTranscoder transcoder, IAssetSettings settings) + private static void WriteSettings(PipeWriter writer, IAssetTranscoder encoder, TSettings settings) + where TAsset : class { var buffer = new ArrayBufferWriter(); - transcoder.WriteSettings(settings, buffer); + encoder.WriteSettings(settings, buffer); writer.Write(buffer.WrittenCount); writer.Write(buffer.WrittenSpan); } - private static async Task WritePayload(PipeWriter writer, AssetId id, IAssetSettings settings, IAssetTranscoder encoder, VirtualFileSystemSpy spy) + private static async Task WritePayload(PipeWriter writer, AssetId id, IAssetTranscoder encoder, TSettings settings, VirtualFileSystemSpy spy) + where TAsset : class { var payload = new ArrayBufferWriter(); await encoder.Encode(id, settings, spy, payload); diff --git a/source/CapriKit.AssetPipeline/AssetFileCache.cs b/source/CapriKit.AssetPipeline/AssetFileCache.cs deleted file mode 100644 index 761e00d..0000000 --- a/source/CapriKit.AssetPipeline/AssetFileCache.cs +++ /dev/null @@ -1,58 +0,0 @@ -using CapriKit.IO; -using System.Buffers; - -namespace CapriKit.AssetPipeline; - -internal sealed class AssetFileCache(IVirtualFileSystem FileSystem, TranscoderCollection Transcoders) -{ - public async Task> Load(AssetId id, IAssetSettings settings) - where TAsset : class - { - var transcoder = Transcoders.Get(); - - var job = await AssetDecoder.Decode(id, transcoder, FileSystem); - return job.Match( - (_, asset) => - { - if (IsUpToDate(asset) && SettingsEqual(transcoder, asset.Settings, settings)) - { - return job; - } - return AssetJob.Missing(id); - }, - (_, _) => job, - (_) => job - ); - } - - private bool IsUpToDate(Asset asset) - where T : class - { - foreach (var (file, version) in asset.Dependencies) - { - if (!FileSystem.Exists(file)) - { - return false; - } - - var lastWrite = FileSystem.LastWriteTime(file); - if (version < lastWrite) - { - return false; - } - } - - return true; - } - - private static bool SettingsEqual(IAssetTranscoder transcoder, IAssetSettings embedded, IAssetSettings requested) - { - var embeddedWriter = new ArrayBufferWriter(); - transcoder.WriteSettings(embedded, embeddedWriter); - - var requestedWriter = new ArrayBufferWriter(); - transcoder.WriteSettings(requested, requestedWriter); - - return embeddedWriter.WrittenSpan.SequenceEqual(requestedWriter.WrittenSpan); - } -} diff --git a/source/CapriKit.AssetPipeline/AssetManager.cs b/source/CapriKit.AssetPipeline/AssetManager.cs index 4df68a6..28fc005 100644 --- a/source/CapriKit.AssetPipeline/AssetManager.cs +++ b/source/CapriKit.AssetPipeline/AssetManager.cs @@ -1,113 +1,130 @@ -using CapriKit.AssetPipeline.HotReloading; +using CapriKit.AssetPipeline.v2; using CapriKit.IO; using Microsoft.Extensions.Logging; +using System.Buffers; namespace CapriKit.AssetPipeline; -/// -/// Encodes, decodes, loads and tracks assets. -/// -public sealed class AssetManager +public sealed partial class AssetManager : IDisposable { + private readonly ILogger Logger; private readonly IVirtualFileSystem FileSystem; - private readonly TranscoderCollection Transcoders; + private readonly AssetCache Cache; + private readonly HotReloadManager HotReloadManager; - private readonly AssetMemoryCache Cache; - private readonly AssetFileCache FileCache; - private readonly HotSwapManager HotSwapManager; - - public AssetManager(ILoggerFactory logger, DirectoryPath rootDirectory) - : this(logger, new FileSystem().ScopedTo(rootDirectory)) { } public AssetManager(ILoggerFactory logger, IVirtualFileSystem fileSystem) { + Logger = logger.CreateLogger(); FileSystem = fileSystem; - Transcoders = new TranscoderCollection(); - Cache = new AssetMemoryCache(); - FileCache = new AssetFileCache(fileSystem, Transcoders); - HotSwapManager = new HotSwapManager(logger, this, fileSystem); + Cache = new AssetCache(); + HotReloadManager = new HotReloadManager(logger, fileSystem); } - /// - public void PushScope() => Cache.PushScope(); + public async Task Load(AssetId id, IAssetTranscoder transcoder, TSettings settings) + where TAsset : class + { + // Check if the asset was loaded before + if (Cache.TryLease(id, out var cachedAsset)) + { + LogLoadedFromCache(Logger, id); + return cachedAsset; + } - /// - public void PopScope() => Cache.PopScope(); + // If not, check if it can be loaded from an up-to-date build + var build = await AssetDecoder.TryDecodeBuildMetaData(id, transcoder, FileSystem); + if (build != default && IsUpToDate(transcoder, settings, build)) + { + var upToDateAsset = await AssetDecoder.Decode(id, transcoder, FileSystem); + RegisterAsset(upToDateAsset, transcoder); - /// - public void RegisterTranscoder(IAssetTranscoder transcoder) - { - Transcoders.Register(transcoder); - } + LogLoadedFromFile(Logger, id); + return upToDateAsset.Value; + } - public Task Encode(AssetId id, IAssetSettings settings) - { - return AssetEncoder.Encode(id, settings, Transcoders.Get(), FileSystem); + // If not, try to rebuild and load the asset + if (!FileSystem.Exists(id.Path)) + { + throw new FileNotFoundException("Could not find primary file to build asset from", id.Path); + } + + await AssetEncoder.Encode(id, transcoder, settings, FileSystem); + var freshAsset = await AssetDecoder.Decode(id, transcoder, FileSystem); + RegisterAsset(freshAsset, transcoder); + + LogBuildAndLoaded(Logger, id); + return freshAsset.Value; } - public Task Encode(AssetId id) - where TAsset : class + public void Unload(AssetId id) { - return Encode(id, default(NoSettings)); + Cache.Return(id); } - /// - /// Immediately decodes an asset, bypasses cache and hot reloading mechanisms. - /// - public Task> Decode(AssetId id) - where TAsset : class + /// Must be called from the main thread. + public void Update() { - return AssetDecoder.Decode(id, Transcoders.Get(), FileSystem); + Cache.Collect(); + HotReloadManager.Update(); } - /// - /// Loads an asset from the cache, decoding it from disk and building it first if it is missing or - /// out of date. The first time an asset is loaded it is put in the current scope, see . - /// - public async Task Load(AssetId id, IAssetSettings settings) + private bool IsUpToDate(IAssetTranscoder transcoder, TSettings settings, AssetBuildMetaData build) where TAsset : class { - // Live asset: ready for use. - if (Cache.TryGet(id, out var entry)) + // Transcoders differ + if (transcoder.Id != build.TranscoderId || transcoder.Version != build.TranscoderVersion) { - return entry; + return false; } - // Matching encoded version available on disk: decode, register for hot swapping and return. - var getFromFileCache = await FileCache.Load(id, settings); - if (getFromFileCache.OnSuccess(out var cachedAsset)) - { - Cache.Add(id, cachedAsset.Value); - HotSwapManager.Track(cachedAsset, settings); - return cachedAsset.Value; - } + // Settings differ + var inUse = new ArrayBufferWriter(); + transcoder.WriteSettings(settings, inUse); - if (getFromFileCache.OnFailure(out var exception)) - { - // TODO: log - } + var inFile = new ArrayBufferWriter(); + transcoder.WriteSettings(build.Settings, inFile); - // Encoded version unavailable, out of date or created using different settings: - // Encode, decode, register for hot swapping and return. - await Encode(id, settings); - var getFromFullBuild = await Decode(id); - if (getFromFullBuild.OnSuccess(out var freshAsset)) + if (!inUse.WrittenSpan.SequenceEqual(inFile.WrittenSpan)) { - Cache.Add(id, freshAsset.Value); - HotSwapManager.Track(freshAsset, settings); + return false; } - if (getFromFileCache.OnFailure(out var rebuildFailure)) + // Dependencies have changed + foreach (var (file, version) in build.Dependencies) { - rebuildFailure.Throw(); + if (!FileSystem.Exists(file)) + { + return false; + } + + var lastWrite = FileSystem.LastWriteTime(file); + if (version < lastWrite) + { + return false; + } } - throw new Exception($"Asset {id} could not be found"); + return true; } - internal void HotSwap(TAsset instance, TAsset replacement) + private void RegisterAsset(Asset asset, IAssetTranscoder transcoder) + where TAsset : class { - var transcoder = Transcoders.Get(); - transcoder.HotSwap(instance, replacement); + Cache.Put(asset.Id, asset.Value); + HotReloadManager.Track(asset, transcoder); } + + public void Dispose() + { + Cache.Dispose(); + } + + [LoggerMessage(Level = LogLevel.Information, Message = "Loaded asset: {asset} from cache")] + private static partial void LogLoadedFromCache(ILogger logger, AssetId asset); + + [LoggerMessage(Level = LogLevel.Information, Message = "Loaded up-to-date asset: {asset} from file")] + private static partial void LogLoadedFromFile(ILogger logger, AssetId asset); + + [LoggerMessage(Level = LogLevel.Information, Message = "Build and loaded fresh asset: {asset}")] + private static partial void LogBuildAndLoaded(ILogger logger, AssetId asset); } diff --git a/source/CapriKit.AssetPipeline/AssetMemoryCache.cs b/source/CapriKit.AssetPipeline/AssetMemoryCache.cs deleted file mode 100644 index e60b826..0000000 --- a/source/CapriKit.AssetPipeline/AssetMemoryCache.cs +++ /dev/null @@ -1,88 +0,0 @@ -using System.Diagnostics.CodeAnalysis; - -namespace CapriKit.AssetPipeline; - -/// -/// Caches assets in stacked scopes that can be discarded in one go. -/// Assets are released when the scope they were added in is popped or this class is disposed -/// -public sealed class AssetMemoryCache : IDisposable -{ - private record CacheItem(int Scope, object Item, IDisposable? Disposable); - - private readonly Dictionary Cache; - private int scope; - - public AssetMemoryCache() - { - scope = 0; - Cache = []; - } - - /// - /// Opens a new scope. Assets added from now on are discarded by the matching . - /// - public void PushScope() - { - scope++; - } - - /// - /// Adds an asset to the current scope. - /// - /// Adding an item with the same id twice throws - public void Add(AssetId id, T asset) - where T : class - { - if (Cache.ContainsKey(id)) - { - throw new InvalidOperationException($"Cannot add item with id: {id} a second time"); - } - Cache[id] = new CacheItem(scope, asset, asset as IDisposable); - } - - public bool TryGet(AssetId id, [NotNullWhen(true)] out T? asset) - where T : class - { - if (Cache.TryGetValue(id, out var entry)) - { - asset = (T)entry.Item; - return true; - } - - asset = default; - return false; - } - - /// - /// Pops the current scope, discarding and disposing every asset added within it. - /// - public void PopScope() - { - if (scope <= 0) - { - throw new InvalidOperationException("No scope to pop"); - } - - foreach (var (key, value) in Cache) - { - if (value.Scope >= scope) - { - value.Disposable?.Dispose(); - Cache.Remove(key); - } - } - - scope--; - } - - public void Dispose() - { - foreach (var (_, value) in Cache) - { - value.Disposable?.Dispose(); - } - - Cache.Clear(); - } -} diff --git a/source/CapriKit.AssetPipeline/v2/HotReloadManager.cs b/source/CapriKit.AssetPipeline/HotReloadManager.cs similarity index 93% rename from source/CapriKit.AssetPipeline/v2/HotReloadManager.cs rename to source/CapriKit.AssetPipeline/HotReloadManager.cs index 9aeb4f0..5e1a5e5 100644 --- a/source/CapriKit.AssetPipeline/v2/HotReloadManager.cs +++ b/source/CapriKit.AssetPipeline/HotReloadManager.cs @@ -18,12 +18,12 @@ internal sealed partial class HotReloadManager : IDisposable private readonly ILogger Logger; private readonly IVirtualFileSystem FileSystem; + private readonly IVirtualFileSystemWatcher Watcher; + private readonly FileSystemEventQueue FileChances; private readonly Dictionary Tracked; private readonly Dictionary> Dependents; - private readonly IVirtualFileSystemWatcher Watcher; - private readonly FileSystemEventQueue FileChances; private readonly HashSet PendingRebuilds; private readonly ConcurrentQueue PendingReloads; @@ -33,13 +33,14 @@ internal sealed partial class HotReloadManager : IDisposable public HotReloadManager(ILoggerFactory logger, IVirtualFileSystem fileSystem) { Logger = logger.CreateLogger(); + FileSystem = fileSystem; + Watcher = fileSystem.Watch(DirectoryPath.Empty); // Watch for all changed, usually fileSystem is a ScopedVirtualFileSystem + FileChances = new FileSystemEventQueue(Watcher); Tracked = []; Dependents = []; - Watcher = fileSystem.Watch(DirectoryPath.Empty); // Watch for all changed, usually fileSystem is a ScopedVirtualFileSystem - FileChances = new FileSystemEventQueue(Watcher); PendingRebuilds = []; PendingReloads = []; @@ -98,6 +99,7 @@ private void DrainFileChanges() } } } + private void ReloadOne() { if (PendingRebuilds.Count > 0) @@ -111,10 +113,15 @@ private void ReloadOne() var reloadable = Tracked[id]; reloadable.Reload(FileSystem, PendingReloads) - .FireAndForget(ex => LogReloadFailed(Logger, id, ex), () => + .FireAndForget(ex => { + LogReloadFailed(Logger, id, ex); isReloading = false; + }, + () => + { LogReloadCompleted(Logger, id); + isReloading = false; }); } } @@ -148,7 +155,6 @@ public void Dispose() [LoggerMessage(Level = LogLevel.Information, Message = "Detected file change: {path}, affecting asset: {asset}")] private static partial void LogPendingReload(ILogger logger, FilePath path, AssetId asset); - [LoggerMessage(Level = LogLevel.Information, Message = "Reloading asset started: {asset}")] private static partial void LogReloadStarted(ILogger logger, AssetId asset); diff --git a/source/CapriKit.AssetPipeline/v2/HotReloadable.cs b/source/CapriKit.AssetPipeline/HotReloadable.cs similarity index 62% rename from source/CapriKit.AssetPipeline/v2/HotReloadable.cs rename to source/CapriKit.AssetPipeline/HotReloadable.cs index 6f91b2f..ab44d99 100644 --- a/source/CapriKit.AssetPipeline/v2/HotReloadable.cs +++ b/source/CapriKit.AssetPipeline/HotReloadable.cs @@ -1,7 +1,7 @@ using CapriKit.IO; using System.Collections.Concurrent; -namespace CapriKit.AssetPipeline.v2; +namespace CapriKit.AssetPipeline; internal sealed record HotSwapAction(AssetId Id, Action PerformHotSwap); @@ -14,14 +14,15 @@ internal abstract class HotReloadable(AssetId id) internal sealed class HotReloadable : HotReloadable where TAsset : class { - private readonly WeakReference> Instance; + private readonly WeakReference Instance; + private readonly TSettings Settings; private readonly IAssetTranscoder Transcoder; - public HotReloadable(Asset asset, IAssetTranscoder transcoder) : base(asset.Id) { - Instance = new WeakReference>(asset); + Instance = new WeakReference(asset.Value); + Settings = asset.BuildMetaData.Settings; Transcoder = transcoder; } @@ -29,9 +30,9 @@ public override async Task Reload(IVirtualFileSystem fileSystem, ConcurrentQueue { if (!Instance.TryGetTarget(out var cold)) { return; } - await AssetEncoder.Encode(cold.Id, Transcoder, cold.BuildMetaData.Settings, fileSystem); - var hot = await AssetDecoder.Decode(cold.Id, Transcoder, fileSystem); + await AssetEncoder.Encode(Id, Transcoder, Settings, fileSystem); + var hot = await AssetDecoder.Decode(Id, Transcoder, fileSystem); - hotSwapActionQueue.Enqueue(new HotSwapAction(cold.Id, () => Transcoder.HotSwap(cold.Value, hot.Value))); + hotSwapActionQueue.Enqueue(new HotSwapAction(Id, () => Transcoder.HotSwap(cold, hot.Value))); } } diff --git a/source/CapriKit.AssetPipeline/HotReloading/HotSwapManager.cs b/source/CapriKit.AssetPipeline/HotReloading/HotSwapManager.cs deleted file mode 100644 index 9260c04..0000000 --- a/source/CapriKit.AssetPipeline/HotReloading/HotSwapManager.cs +++ /dev/null @@ -1,137 +0,0 @@ -using CapriKit.Concurrency.Async; -using CapriKit.IO; -using CapriKit.IO.Watchers; -using Microsoft.Extensions.Logging; -using System.Collections.Concurrent; -using System.Diagnostics; - -namespace CapriKit.AssetPipeline.HotReloading; - -/// -/// Facilitates hot reloading and hot swapping of assets. Tracks the files used to create an asset -/// and triggers a rebuild on file changes. Takes care of threading and only performs the final -/// hot swap when is called from the main thread. -/// -internal sealed partial class HotSwapManager : IDisposable -{ - private static readonly TimeSpan MinWaitTime = TimeSpan.FromSeconds(0.5); - - private readonly ILogger Logger; - - private readonly AssetManager AssetManager; - private readonly IReadOnlyVirtualFileSystem FileSystem; - - private readonly Dictionary Tracked; - private readonly Dictionary> Dependents; - - private readonly IVirtualFileSystemWatcher Watcher; - private readonly FileSystemEventQueue PendingFileChanges; - private readonly HashSet PendingReloads; - private readonly ConcurrentQueue PendingHotSwaps; - - private long lastFileChange; - - public HotSwapManager(ILoggerFactory logger, AssetManager assetManager, IReadOnlyVirtualFileSystem fileSystem) - { - Logger = logger.CreateLogger(); - AssetManager = assetManager; - FileSystem = fileSystem; - Tracked = []; - Dependents = []; - - Watcher = fileSystem.Watch(DirectoryPath.Empty); // Watch for all changes, usually fileSystem is a ScopedVirtualFileSystem - PendingFileChanges = new FileSystemEventQueue(Watcher); - PendingReloads = []; - PendingHotSwaps = []; - lastFileChange = Stopwatch.GetTimestamp(); - } - - public void Track(Asset asset, IAssetSettings settings) - where TAsset : class - { - Tracked[asset.Id] = new Reloadable(asset.Id, asset.Value, settings); - foreach (var dependency in asset.Dependencies) - { - var file = dependency.File; - if (!Dependents.TryGetValue(file, out var ids)) { ids = Dependents[file] = []; } - ids.Add(asset.Id); - } - } - - public void ProcessUpdates() - { - DrainFileChanges(); - var elapsed = Stopwatch.GetElapsedTime(lastFileChange); - if (elapsed > MinWaitTime) - { - ReloadAffected(); - } - - HotSwapCompleted(); - } - - private void DrainFileChanges() - { - while (PendingFileChanges.TryDequeue(out var @event)) - { - if (Dependents.TryGetValue(@event.File, out var dependents)) - { - lastFileChange = Stopwatch.GetTimestamp(); - foreach (var assetId in dependents) - { - LogPendingReload(Logger, @event.File, assetId); - PendingReloads.Add(assetId); - } - } - } - } - - private void ReloadAffected() - { - PendingReloads.RemoveWhere(id => - { - var reloadable = Tracked[id]; - // Prevent kicking off a reload while the asset is still being reloaded - if (reloadable.IsReloading) { return false; } - - LogReloadStarted(Logger, id); - reloadable.Reload(AssetManager, PendingHotSwaps).FireAndForget( - ex => LogReloadFailed(Logger, id, ex)); - - return true; - }); - } - - private void HotSwapCompleted() - { - while (PendingHotSwaps.TryDequeue(out var result)) - { - try - { - result.HotSwap(AssetManager, this); - LogReloadCompleted(Logger, result.Id); - } - catch (Exception ex) - { - LogReloadFailed(Logger, result.Id, ex); - } - } - } - - public void Dispose() - { - Watcher.Stop(); - } - - [LoggerMessage(Level = LogLevel.Information, Message = "Detected file change: {path}, affecting asset: {asset}")] - private static partial void LogPendingReload(ILogger logger, FilePath path, AssetId asset); - - [LoggerMessage(Level = LogLevel.Information, Message = "Reloading asset started: {asset}")] - private static partial void LogReloadStarted(ILogger logger, AssetId asset); - - [LoggerMessage(Level = LogLevel.Information, Message = "Reloading asset completed: {asset}")] - private static partial void LogReloadCompleted(ILogger logger, AssetId asset); - - [LoggerMessage(Level = LogLevel.Error, Message = "Reloading asset failed: {asset}")] - private static partial void LogReloadFailed(ILogger logger, AssetId asset, Exception exception); -} diff --git a/source/CapriKit.AssetPipeline/HotReloading/HotSwappable.cs b/source/CapriKit.AssetPipeline/HotReloading/HotSwappable.cs deleted file mode 100644 index 310a60a..0000000 --- a/source/CapriKit.AssetPipeline/HotReloading/HotSwappable.cs +++ /dev/null @@ -1,34 +0,0 @@ -namespace CapriKit.AssetPipeline.HotReloading; - -/// -/// Represents an asset that is ready to be hot swapped by the main thread -/// -internal abstract class HotSwappable(AssetId id) -{ - public AssetId Id { get; } = id; - public abstract void HotSwap(AssetManager manager, HotSwapManager hotSwapManager); -} - -internal sealed class HotSwappable(TAsset instance, AssetJob newParts, IAssetSettings settings) - : HotSwappable(newParts.Id) - where TAsset : class -{ - public override void HotSwap(AssetManager assetManager, HotSwapManager hotSwapManager) - { - if (newParts.OnSuccess(out var asset)) - { - assetManager.HotSwap(instance, asset.Value); - - // Instance keeps being the active object, so keep tracking instance, but with the new dependencies - var toTrack = new Asset(asset.Id, instance, settings, asset.Dependencies); - hotSwapManager.Track(toTrack, settings); - } - - if (newParts.OnFailure(out var ex)) - { - ex.Throw(); - } - - throw new Exception($"Asset {Id} tracked for hot swapping could no longer be found"); - } -} diff --git a/source/CapriKit.AssetPipeline/HotReloading/Reloadable.cs b/source/CapriKit.AssetPipeline/HotReloading/Reloadable.cs deleted file mode 100644 index 4d96099..0000000 --- a/source/CapriKit.AssetPipeline/HotReloading/Reloadable.cs +++ /dev/null @@ -1,51 +0,0 @@ -using System.Collections.Concurrent; - -namespace CapriKit.AssetPipeline.HotReloading; - -/// -/// Represents an asset that can be rebuild and reloaded on demand. -/// -internal abstract class Reloadable -{ - protected volatile bool isReloading; - - public abstract AssetId Id { get; } - public abstract Task Reload(AssetManager manager, ConcurrentQueue queue); - - public bool IsReloading => isReloading; -} - -internal sealed class Reloadable : Reloadable - where TAsset : class -{ - private readonly WeakReference Instance; - private readonly IAssetSettings Settings; - - public Reloadable(AssetId id, TAsset asset, IAssetSettings settings) - { - Instance = new WeakReference(asset); - Settings = settings; - Id = id; - } - - public override AssetId Id { get; } - - public override async Task Reload(AssetManager manager, ConcurrentQueue queue) - { - try - { - isReloading = true; - if (!Instance.TryGetTarget(out var cold)) { return; } - - // TODO: technically cold can be alive but disposed here - - await manager.Encode(Id, Settings); - var hot = await manager.Decode(Id); - queue.Enqueue(new HotSwappable(cold, hot, Settings)); - } - finally - { - isReloading = false; - } - } -} diff --git a/source/CapriKit.AssetPipeline/IAssetTranscoder.cs b/source/CapriKit.AssetPipeline/IAssetTranscoder.cs index 6985e61..51a7a68 100644 --- a/source/CapriKit.AssetPipeline/IAssetTranscoder.cs +++ b/source/CapriKit.AssetPipeline/IAssetTranscoder.cs @@ -3,77 +3,58 @@ namespace CapriKit.AssetPipeline; -public readonly struct NoSettings : IAssetSettings { } - -public interface IAssetSettings { } - +/// +/// Interface for classes that builds assets (such as texture, models and sound effects) and load them +/// when the program needs them. +/// public interface IAssetTranscoder { Guid Id { get; } int Version { get; } } -// The pipeline consumes transcoders through this settings-erased view so that AssetManager, -// AssetEncoder and AssetDecoder never need a TSettings type parameter. The erased members are -// internal because only the pipeline should call them; transcoder authors implement -// IAssetTranscoder, which bridges them. -public interface IAssetTranscoder : IAssetTranscoder +/// +public interface IAssetTranscoder : IAssetTranscoder + where TAsset : class { - internal Task Encode(AssetId id, IAssetSettings settings, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer); - internal TAsset Decode(AssetId id, IAssetSettings settings, ref SequenceReader reader); - internal IAssetSettings ReadSettings(ref SequenceReader reader); - internal void WriteSettings(IAssetSettings settings, IBufferWriter writer); + // Asynchronous since we expect the encoder to read external files /// - /// Moves the contents of into . - /// keeps its identity and stays the live object that callers - /// already hold references to. The transcoder is responsible for cleaning-up any - /// orphaned resources. After calling this method must no longer - /// be used or referenced. + /// Loads the raw asset data from the file system and build/encodes it into a format optimized for loading and handling in + /// an interactive simulation. Encoding happens asynchronously and can happen on any thread. /// - void HotSwap(TAsset instance, TAsset newParts); -} - -public interface IAssetTranscoder : IAssetTranscoder - where TSettings : IAssetSettings -{ - // Asynchronous since we expect the encoder to read external files - Task Encode(AssetId id, TSettings settings, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer); + public Task Encode(AssetId id, TSettings settings, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer); // Synchronous by design: the envelope owns all file IO and hands the decoder an // in-memory payload. The reader's buffer is only valid for the duration of the call, // decoders must copy out anything they want to keep. - TAsset Decode(AssetId id, TSettings settings, ref SequenceReader reader); - - // `new` because it hides the erased ReadSettings: same parameters, more specific return type - new TSettings ReadSettings(ref SequenceReader reader); - - void WriteSettings(TSettings settings, IBufferWriter writer); - // Default implementations that bridge the settings-erased members to their typed - // counterparts. This is the only place where the pipeline transitions from - // IAssetSettings back to the concrete TSettings - Task IAssetTranscoder.Encode(AssetId id, IAssetSettings settings, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer) - => Encode(id, AsTypedSettings(settings), fileSystem, writer); + /// + /// Decodes the file created the into an object by reading bytes from the given reader. Though decoding + /// itself is synchronous, it can happen as part of a multi-threaded or async operation. + /// + public TAsset Decode(AssetId id, TSettings settings, ref SequenceReader reader); - TAsset IAssetTranscoder.Decode(AssetId id, IAssetSettings settings, ref SequenceReader reader) - => Decode(id, AsTypedSettings(settings), ref reader); - IAssetSettings IAssetTranscoder.ReadSettings(ref SequenceReader reader) - => ReadSettings(ref reader); + /// + /// Encodes the settings required to encode/decode the asset into the stream. Though this is by + /// itself synchronous, it can happen as part of a multi-threaded or async operation. + /// + public void WriteSettings(TSettings settings, IBufferWriter writer); - void IAssetTranscoder.WriteSettings(IAssetSettings settings, IBufferWriter writer) - => WriteSettings(AsTypedSettings(settings), writer); - private static TSettings AsTypedSettings(IAssetSettings settings) - { - if (settings is not TSettings typedSettings) - { - throw new ArgumentException( - $"Transcoder for {typeof(TAsset).Name} expects settings of type {typeof(TSettings).Name} but got {settings?.GetType().Name ?? "null"}", - nameof(settings)); - } + /// + /// Decodes the settings required to encode/decode the asset into the stream. Though this is by + /// itself synchronous, it can happen as part of a multi-threaded or async operation. + /// + public TSettings ReadSettings(ref SequenceReader reader); - return typedSettings; - } + /// + /// Moves the contents of into . + /// keeps its identity and stays the live object that callers + /// already hold references to. The transcoder is responsible for cleaning-up any + /// orphaned resources. After calling this method must no longer + /// be used or referenced. + /// + void HotSwap(TAsset instance, TAsset newParts); } diff --git a/source/CapriKit.AssetPipeline/NoSettingsTranscoder.cs b/source/CapriKit.AssetPipeline/NoSettingsTranscoder.cs index d0a1e24..a86ec8c 100644 --- a/source/CapriKit.AssetPipeline/NoSettingsTranscoder.cs +++ b/source/CapriKit.AssetPipeline/NoSettingsTranscoder.cs @@ -3,22 +3,30 @@ namespace CapriKit.AssetPipeline; -public abstract class NoSettingsTranscoder(Guid id, int version) : IAssetTranscoder> +public readonly struct NoSettings; + +/// +/// Abstract transcoder that removes some of the boilerplate code for implementations that do not have any settings. +/// +public abstract class NoSettingsTranscoder(Guid id, int version) : IAssetTranscoder + where TAsset : class { public Guid Id { get; } = id; public int Version { get; } = version; - public abstract TAsset Decode(AssetId id, NoSettings settings, ref SequenceReader reader); - public abstract Task Encode(AssetId id, NoSettings settings, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer); + public abstract Task Encode(AssetId id, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer); + + public abstract TAsset Decode(AssetId id, ref SequenceReader reader); + public abstract void HotSwap(TAsset instance, TAsset newParts); - public NoSettings ReadSettings(ref SequenceReader reader) - { - return default; - } + TAsset IAssetTranscoder.Decode(AssetId id, NoSettings settings, ref SequenceReader reader) + => Decode(id, ref reader); + + Task IAssetTranscoder.Encode(AssetId id, NoSettings settings, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer) + => Encode(id, fileSystem, writer); + + NoSettings IAssetTranscoder.ReadSettings(ref SequenceReader reader) => default; - public void WriteSettings(NoSettings settings, IBufferWriter writer) - { - // no-op - } + void IAssetTranscoder.WriteSettings(NoSettings settings, IBufferWriter writer) { } } diff --git a/source/CapriKit.AssetPipeline/ServiceCollectionExtensions.cs b/source/CapriKit.AssetPipeline/ServiceCollectionExtensions.cs index ab51c87..4f91264 100644 --- a/source/CapriKit.AssetPipeline/ServiceCollectionExtensions.cs +++ b/source/CapriKit.AssetPipeline/ServiceCollectionExtensions.cs @@ -12,7 +12,8 @@ public static IServiceCollection AddAssetPipeline(this IServiceCollection servic return services.AddSingleton(sp => { var logFactory = sp.GetRequiredService(); - return new AssetManager(logFactory, assetDirectory); + var fileSystem = new FileSystem().ScopedTo(assetDirectory); + return new AssetManager(logFactory, fileSystem); }); } } diff --git a/source/CapriKit.AssetPipeline/TranscoderCollection.cs b/source/CapriKit.AssetPipeline/TranscoderCollection.cs deleted file mode 100644 index 5ed865d..0000000 --- a/source/CapriKit.AssetPipeline/TranscoderCollection.cs +++ /dev/null @@ -1,21 +0,0 @@ -namespace CapriKit.AssetPipeline; - -internal sealed class TranscoderCollection -{ - private readonly Dictionary Transcoders = []; - - public void Register(IAssetTranscoder transcoder) - { - Transcoders[typeof(TAsset)] = transcoder; - } - - public IAssetTranscoder Get() - { - if (!Transcoders.TryGetValue(typeof(TAsset), out var transcoder)) - { - throw new InvalidOperationException($"No transcoder registered for asset type {typeof(TAsset).Name}"); - } - - return (IAssetTranscoder)transcoder; - } -} diff --git a/source/CapriKit.AssetPipeline/v2/Asset.cs b/source/CapriKit.AssetPipeline/v2/Asset.cs deleted file mode 100644 index c9f35a2..0000000 --- a/source/CapriKit.AssetPipeline/v2/Asset.cs +++ /dev/null @@ -1,17 +0,0 @@ -using CapriKit.IO; - -namespace CapriKit.AssetPipeline.v2; - -/// -/// Unique asset identifier -/// -/// Optional key to a sub-resources in Path. -/// Virtual file path that points to the file the asset originates from. -public record AssetId(string Key, FilePath Path); - -public record Asset(AssetId Id, TAsset Value, AssetBuildMetaData BuildMetaData) - where TAsset : class; - -public record class AssetBuildMetaData(Guid TranscoderId, int TranscoderVersion, TSettings Settings, IReadOnlyList Dependencies); - -public sealed record Dependency(FilePath File, DateTime Version); diff --git a/source/CapriKit.AssetPipeline/v2/AssetDecoder.cs b/source/CapriKit.AssetPipeline/v2/AssetDecoder.cs deleted file mode 100644 index d059968..0000000 --- a/source/CapriKit.AssetPipeline/v2/AssetDecoder.cs +++ /dev/null @@ -1,136 +0,0 @@ -using CapriKit.IO; -using CapriKit.IO.Streams; -using System.Buffers; -using static CapriKit.AssetPipeline.v2.AssetUtilities; - -namespace CapriKit.AssetPipeline.v2; - -/// -/// Decodes the generic asset envelope and, using a specialized IAssetTranscoder, the asset itself -/// -internal static class AssetDecoder -{ - public static async Task> Decode(AssetId id, IAssetTranscoder decoder, IReadOnlyVirtualFileSystem fileSystem) - where TAsset : class - { - var inputPath = ToEncodedFilePath(id); - if (!fileSystem.Exists(inputPath)) - { - throw new FileNotFoundException($"Could not find file: {inputPath} to load asset: {id}", id.Path); - } - - using var input = fileSystem.OpenRead(inputPath); - var length = checked((int)input.Length); - var buffer = ArrayPool.Shared.Rent(length); - - try - { - await input.ReadExactlyAsync(buffer.AsMemory(0, length)); - var reader = SequenceReaders.Create(buffer, 0, length); - - var (encoderId, encoderVersion) = ReadHeader(ref reader); - ThrowOnDecoderMismatch(id, inputPath, encoderId, encoderVersion, decoder); - - var settings = ReadSettings(ref reader, decoder); - var asset = ReadPayload(ref reader, id, decoder, settings); - var dependencies = ReadDependencies(ref reader); - - var buildMetaData = new AssetBuildMetaData(encoderId, encoderVersion, settings, dependencies); - return new Asset(id, asset, buildMetaData); - } - finally - { - ArrayPool.Shared.Return(buffer); - } - } - - public static async Task?> TryDecodeBuildMetaData(AssetId id, IAssetTranscoder decoder, IReadOnlyVirtualFileSystem fileSystem) - where TAsset : class - { - try - { - var inputPath = ToEncodedFilePath(id); - if (!fileSystem.Exists(inputPath)) - { - throw new FileNotFoundException($"Could not find file: {inputPath} to load asset: {id}", id.Path); - } - - using var input = fileSystem.OpenRead(inputPath); - var length = checked((int)input.Length); - var buffer = ArrayPool.Shared.Rent(length); - - try - { - await input.ReadExactlyAsync(buffer.AsMemory(0, length)); - var reader = SequenceReaders.Create(buffer, 0, length); - - var (encoderId, encoderVersion) = ReadHeader(ref reader); - var settings = ReadSettings(ref reader, decoder); - SkipPayload(ref reader); - var dependencies = ReadDependencies(ref reader); - return new AssetBuildMetaData(encoderId, encoderVersion, settings, dependencies); - } - finally - { - ArrayPool.Shared.Return(buffer); - } - } - catch - { - return null; - } - } - - private static (Guid id, int version) ReadHeader(ref SequenceReader reader) - { - var id = reader.ReadGuid(); - var version = reader.ReadInt32(); - return (id, version); - } - - private static TSettings ReadSettings(ref SequenceReader reader, IAssetTranscoder decoder) - where TAsset : class - { - var settingsLength = reader.ReadInt32(); - var settingsReader = reader.SliceUnread(settingsLength); - return decoder.ReadSettings(ref settingsReader); - } - - private static TAsset ReadPayload(ref SequenceReader reader, AssetId id, IAssetTranscoder decoder, TSettings settings) - where TAsset : class - { - var payloadLength = reader.ReadInt32(); - var payloadReader = reader.SliceUnread(payloadLength); - return decoder.Decode(id, settings, ref payloadReader); - } - - private static void SkipPayload(ref SequenceReader reader) - { - var payloadLength = reader.ReadInt32(); - reader.Advance(payloadLength); - } - - private static List ReadDependencies(ref SequenceReader reader) - { - var count = reader.ReadInt32(); - var dependencies = new List(count); - for (var i = 0; i < count; i++) - { - var lastWriteTicks = reader.ReadInt64(); - var lastWrite = new DateTime(lastWriteTicks); - var filePathString = reader.ReadString(); - var filePath = new FilePath(filePathString); - dependencies.Add(new Dependency(filePath, lastWrite)); - } - return dependencies; - } - - private static void ThrowOnDecoderMismatch(AssetId id, FilePath file, Guid fileId, int fileVersion, IAssetTranscoder decoder) - { - if (fileId != decoder.Id || fileVersion != decoder.Version) - { - throw new InvalidDataException( - $"Decoder mismatch. Asset: {id} found in file: {file}, was encoded using {fileId}:v{fileVersion} but the current decoder is {decoder.Id}:v{decoder.Version}."); - } - } -} diff --git a/source/CapriKit.AssetPipeline/v2/AssetEncoder.cs b/source/CapriKit.AssetPipeline/v2/AssetEncoder.cs deleted file mode 100644 index fd48902..0000000 --- a/source/CapriKit.AssetPipeline/v2/AssetEncoder.cs +++ /dev/null @@ -1,69 +0,0 @@ -using CapriKit.IO; -using CapriKit.IO.Streams; -using System.Buffers; -using System.IO.Pipelines; -using static CapriKit.AssetPipeline.v2.AssetUtilities; - -namespace CapriKit.AssetPipeline.v2; - -// File format: [encoder id][encoder version][settings length][settings][payload length][payload][dependency count][dependencies] - -/// -/// Encodes the generic asset envelope and, using a specialized IAssetTranscoder, the asset itself -/// -internal static class AssetEncoder -{ - public static async Task Encode(AssetId id, IAssetTranscoder encoder, TSettings settings, IVirtualFileSystem fileSystem) - where TAsset : class - { - ThrowOnFileNotFound(id.Path, fileSystem); - var outputPath = ToEncodedFilePath(id); - - using var output = fileSystem.CreateReadWrite(outputPath); - var writer = PipeWriter.Create(output); - var spy = fileSystem.SpyOn(); - - WriteHeader(writer, encoder); - WriteSettings(writer, encoder, settings); - await WritePayload(writer, id, encoder, settings, spy); - WriteDependencies(writer, spy); - - await writer.FlushAsync(); - await writer.CompleteAsync(); - } - - private static void WriteHeader(PipeWriter writer, IAssetTranscoder encoder) - { - writer.Write(encoder.Id); - writer.Write(encoder.Version); - } - - private static void WriteSettings(PipeWriter writer, IAssetTranscoder encoder, TSettings settings) - where TAsset : class - { - var buffer = new ArrayBufferWriter(); - encoder.WriteSettings(settings, buffer); - writer.Write(buffer.WrittenCount); - writer.Write(buffer.WrittenSpan); - } - - private static async Task WritePayload(PipeWriter writer, AssetId id, IAssetTranscoder encoder, TSettings settings, VirtualFileSystemSpy spy) - where TAsset : class - { - var payload = new ArrayBufferWriter(); - await encoder.Encode(id, settings, spy, payload); - writer.Write(payload.WrittenCount); - writer.Write(payload.WrittenSpan); - } - - private static void WriteDependencies(PipeWriter writer, VirtualFileSystemSpy spy) - { - writer.Write(spy.OpenedFiles.Count); - foreach (var dependency in spy.OpenedFiles) - { - var lastWrite = spy.LastWriteTime(dependency); - writer.Write(lastWrite.Ticks); - writer.Write(dependency); - } - } -} diff --git a/source/CapriKit.AssetPipeline/v2/AssetManager.cs b/source/CapriKit.AssetPipeline/v2/AssetManager.cs deleted file mode 100644 index 1a09fc6..0000000 --- a/source/CapriKit.AssetPipeline/v2/AssetManager.cs +++ /dev/null @@ -1,114 +0,0 @@ -using CapriKit.IO; -using Microsoft.Extensions.Logging; -using System.Buffers; - -namespace CapriKit.AssetPipeline.v2; - -// TODO: Add logging -public sealed class AssetManager : IDisposable -{ - private readonly IVirtualFileSystem FileSystem; - private readonly AssetCache Cache; - private readonly ILogger Logger; - - public AssetManager(ILoggerFactory logger, IVirtualFileSystem fileSystem) - { - Logger = logger.CreateLogger(); - FileSystem = fileSystem; - Cache = new AssetCache(); - } - - public async Task Load(AssetId id, IAssetTranscoder transcoder, TSettings settings) - where TAsset : class - { - // Check if the asset was loaded before - if (Cache.TryLease(id, out var cachedAsset)) - { - return cachedAsset; - } - - // If not, check if it can be loaded from an up-to-date build - var build = await AssetDecoder.TryDecodeBuildMetaData(id, transcoder, FileSystem); - if (build != null && IsUpToDate(transcoder, settings, build)) - { - var upToDateAsset = await AssetDecoder.Decode(id, transcoder, FileSystem); - RegisterAsset(upToDateAsset, transcoder); - return upToDateAsset.Value; - } - - // If not, try to rebuild and load the asset - if (!FileSystem.Exists(id.Path)) - { - throw new FileNotFoundException("Could not find primary file to build asset from", id.Path); - } - - await AssetEncoder.Encode(id, transcoder, settings, FileSystem); - var freshAsset = await AssetDecoder.Decode(id, transcoder, FileSystem); - RegisterAsset(freshAsset, transcoder); - - return freshAsset.Value; - } - - public void Unload(AssetId id) - { - Cache.Return(id); - } - - /// Must be called from the main thread. - public void Update() - { - // TODO: Perform work that can only be done on the main thread (like hot-reloading) - Cache.Collect(); - } - - private bool IsUpToDate(IAssetTranscoder transcoder, TSettings settings, AssetBuildMetaData build) - where TAsset : class - { - // Transcoders differ - if (transcoder.Id != build.TranscoderId || transcoder.Version != build.TranscoderVersion) - { - return false; - } - - // Settings differ - var inUse = new ArrayBufferWriter(); - transcoder.WriteSettings(settings, inUse); - - var inFile = new ArrayBufferWriter(); - transcoder.WriteSettings(build.Settings, inFile); - - if (!inUse.WrittenSpan.SequenceEqual(inFile.WrittenSpan)) - { - return false; - } - - // Dependencies have changed - foreach (var (file, version) in build.Dependencies) - { - if (!FileSystem.Exists(file)) - { - return false; - } - - var lastWrite = FileSystem.LastWriteTime(file); - if (version < lastWrite) - { - return false; - } - } - - return true; - } - - private void RegisterAsset(Asset asset, IAssetTranscoder transcoder) - where TAsset : class - { - // TODO: Register for hot reloading - Cache.Put(asset.Id, asset.Value); - } - - public void Dispose() - { - Cache.Dispose(); - } -} diff --git a/source/CapriKit.AssetPipeline/v2/AssetUtilities.cs b/source/CapriKit.AssetPipeline/v2/AssetUtilities.cs deleted file mode 100644 index f894d16..0000000 --- a/source/CapriKit.AssetPipeline/v2/AssetUtilities.cs +++ /dev/null @@ -1,25 +0,0 @@ -using CapriKit.IO; - -namespace CapriKit.AssetPipeline.v2; - -internal static class AssetUtilities -{ - public static FilePath ToEncodedFilePath(AssetId id) - { - if (string.IsNullOrEmpty(id.Key)) - { - return $"{id.Path}.cka"; - } - - var key = IOUtilities.EscapeFileName(id.Key); - return $"{id.Path}.{key}.cka"; - } - - public static void ThrowOnFileNotFound(FilePath path, IVirtualFileSystem fileSystem) - { - if (!fileSystem.Exists(path)) - { - throw new FileNotFoundException(null, path); - } - } -} diff --git a/source/CapriKit.AssetPipeline/v2/IAssetTranscoder.cs b/source/CapriKit.AssetPipeline/v2/IAssetTranscoder.cs deleted file mode 100644 index 14e5f32..0000000 --- a/source/CapriKit.AssetPipeline/v2/IAssetTranscoder.cs +++ /dev/null @@ -1,37 +0,0 @@ -using CapriKit.IO; -using System.Buffers; - -namespace CapriKit.AssetPipeline.v2; - - -public interface IAssetTranscoder -{ - Guid Id { get; } - int Version { get; } -} - -// TODO: improve documentation -public interface IAssetTranscoder : IAssetTranscoder - where TAsset : class -{ - // Asynchronous since we expect the encoder to read external files - public Task Encode(AssetId id, TSettings settings, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer); - - // Synchronous by design: the envelope owns all file IO and hands the decoder an - // in-memory payload. The reader's buffer is only valid for the duration of the call, - // decoders must copy out anything they want to keep. - public TAsset Decode(AssetId id, TSettings settings, ref SequenceReader reader); - - public void WriteSettings(TSettings settings, IBufferWriter writer); - - public TSettings ReadSettings(ref SequenceReader reader); - - /// - /// Moves the contents of into . - /// keeps its identity and stays the live object that callers - /// already hold references to. The transcoder is responsible for cleaning-up any - /// orphaned resources. After calling this method must no longer - /// be used or referenced. - /// - void HotSwap(TAsset instance, TAsset newParts); -} diff --git a/source/CapriKit.AssetPipeline/v2/NoSettingsTranscoder.cs b/source/CapriKit.AssetPipeline/v2/NoSettingsTranscoder.cs deleted file mode 100644 index 718b9c5..0000000 --- a/source/CapriKit.AssetPipeline/v2/NoSettingsTranscoder.cs +++ /dev/null @@ -1,29 +0,0 @@ -using CapriKit.IO; -using System.Buffers; - -namespace CapriKit.AssetPipeline.v2; - -internal readonly struct NoSettings; - -public abstract class NoSettingsTranscoder(Guid id, int version) : IAssetTranscoder - where TAsset : class -{ - public Guid Id { get; } = id; - public int Version { get; } = version; - - public abstract Task Encode(AssetId id, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer); - - public abstract TAsset Decode(AssetId id, ref SequenceReader reader); - - public abstract void HotSwap(TAsset instance, TAsset newParts); - - TAsset IAssetTranscoder.Decode(AssetId id, NoSettings settings, ref SequenceReader reader) - => Decode(id, ref reader); - - Task IAssetTranscoder.Encode(AssetId id, NoSettings settings, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer) - => Encode(id, fileSystem, writer); - - NoSettings IAssetTranscoder.ReadSettings(ref SequenceReader reader) => default; - - void IAssetTranscoder.WriteSettings(NoSettings settings, IBufferWriter writer) { } -} diff --git a/source/CapriKit.Tests/AssetPipeline/AssetDecoderTests.cs b/source/CapriKit.Tests/AssetPipeline/AssetDecoderTests.cs deleted file mode 100644 index a690336..0000000 --- a/source/CapriKit.Tests/AssetPipeline/AssetDecoderTests.cs +++ /dev/null @@ -1,32 +0,0 @@ -using CapriKit.AssetPipeline; -using CapriKit.IO; - -namespace CapriKit.Tests.AssetPipeline; - -internal class AssetDecoderTests -{ - - [Test] - public async Task Decode() - { - var fileSystem = new InMemoryFileSystem(); - await fileSystem.WriteAllText("hello.txt", "héllo"); - var transcoder = new DummyTranscoder(); - var id = new AssetId("Main", "hello.txt"); - - await AssetEncoder.Encode(id, new NoSettings(), transcoder, fileSystem); - var job = await AssetDecoder.Decode(id, transcoder, fileSystem); - - FilePath expectedDependency = "hello.txt"; - DateTime expectedTimeStamp = DateTime.Now; - - var success = job.OnSuccess(out var asset); - await Assert.That(success).IsTrue(); - - await Assert.That(asset!.Value).IsEqualTo("HÉLLO"); - await Assert.That(asset.Dependencies.Count).IsEqualTo(1); - await Assert.That(asset.Dependencies.First().File).IsEqualTo(expectedDependency); - await Assert.That(asset.Dependencies.First().Version) - .IsBetween(expectedTimeStamp.AddMinutes(-1), expectedTimeStamp.AddMinutes(1)); - } -} diff --git a/source/CapriKit.Tests/AssetPipeline/AssetEncoderTests.cs b/source/CapriKit.Tests/AssetPipeline/AssetEncoderTests.cs deleted file mode 100644 index 8cb0c48..0000000 --- a/source/CapriKit.Tests/AssetPipeline/AssetEncoderTests.cs +++ /dev/null @@ -1,44 +0,0 @@ -using CapriKit.AssetPipeline; -using CapriKit.IO; -using CapriKit.IO.Streams; -using System.Buffers; - -namespace CapriKit.Tests.AssetPipeline; - -internal class AssetEncoderTests -{ - [Test] - public async Task Encode() - { - var fileSystem = new InMemoryFileSystem(); - await fileSystem.WriteAllText("hello.txt", "héllo"); - var transcoder = new DummyTranscoder(); - var id = new AssetId("Main", "hello.txt"); - - await AssetEncoder.Encode(id, new NoSettings(), transcoder, fileSystem); - var bytes = await fileSystem.ReadAllBytes("hello.txt.Main.cka"); - - var reader = new SequenceReader(new ReadOnlySequence(bytes)); - var encoderId = reader.ReadGuid(); - var encoderVersion = reader.ReadInt32(); - var settingsLength = reader.ReadInt32(); - reader.Advance(settingsLength); - var payloadLength = reader.ReadInt32(); - reader.Advance(payloadLength); - var dependencyCount = reader.ReadInt32(); - var lastWrite = reader.ReadInt64(); - var dependency = reader.ReadString(); - var end = reader.End; - - DateTime expectedTimeStamp = DateTime.Now; - - await Assert.That(encoderId).IsEqualTo(transcoder.Id); - await Assert.That(encoderVersion).IsEqualTo(transcoder.Version); - await Assert.That(settingsLength).IsEqualTo(0); - await Assert.That(dependencyCount).IsEqualTo(1); - await Assert.That(new DateTime(lastWrite)) - .IsBetween(expectedTimeStamp.AddMinutes(-1), expectedTimeStamp.AddMinutes(1)); - await Assert.That(dependency).IsEqualTo("hello.txt"); - await Assert.That(end).IsTrue(); - } -} diff --git a/source/CapriKit.Tests/AssetPipeline/AssetManagerTests.cs b/source/CapriKit.Tests/AssetPipeline/AssetManagerTests.cs deleted file mode 100644 index e89985b..0000000 --- a/source/CapriKit.Tests/AssetPipeline/AssetManagerTests.cs +++ /dev/null @@ -1,41 +0,0 @@ -using CapriKit.AssetPipeline; -using CapriKit.IO; -using Microsoft.Extensions.Logging.Abstractions; - -namespace CapriKit.Tests.AssetPipeline; - -internal class AssetManagerTests -{ - [Test] - public async Task Decode() - { - var fileSystem = new InMemoryFileSystem(); - - await fileSystem.WriteAllText("hello.txt", "héllo"); - var manager = new AssetManager(NullLoggerFactory.Instance, fileSystem); - manager.RegisterTranscoder(new DummyTranscoder()); - var id = new AssetId("Main", "hello.txt"); - - await manager.Encode(id); - var text = await manager.Decode(id); - - await Assert.That(text).IsEqualTo("HÉLLO"); - } - - [Test] - public async Task Decode_SettingsAreReadFromTheEncodedFile() - { - var fileSystem = new InMemoryFileSystem(); - - await fileSystem.WriteAllText("hello.txt", "hey"); - var manager = new AssetManager(NullLoggerFactory.Instance, fileSystem); - manager.RegisterTranscoder(new RepeatTranscoder()); - var id = new AssetId("Main", "hello.txt"); - - // The asset type is inferred from the settings, decoding requires no settings at all - await manager.Encode(id, new RepeatSettings(3)); - var text = await manager.Decode(id); - - await Assert.That(text).IsEqualTo("heyheyhey"); - } -} diff --git a/source/CapriKit.Tests/AssetPipeline/DummyTranscoder.cs b/source/CapriKit.Tests/AssetPipeline/DummyTranscoder.cs deleted file mode 100644 index 5619eba..0000000 --- a/source/CapriKit.Tests/AssetPipeline/DummyTranscoder.cs +++ /dev/null @@ -1,43 +0,0 @@ -using CapriKit.AssetPipeline; -using CapriKit.IO; -using CapriKit.IO.Streams; -using System.Buffers; - -namespace CapriKit.Tests.AssetPipeline; - -/// -/// Uppercases a text file -/// -internal sealed class DummyTranscoder : IAssetTranscoder> -{ - public Guid Id => Guid.Parse("{B87F41E3-6C33-46E4-802A-3E1E82800E7A}"); - public int Version => 1; - - public async Task Encode(AssetId id, NoSettings settings, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer) - { - var text = await fileSystem.ReadAllText(id.Path); - writer.Write(text.ToUpperInvariant()); - } - - - - public string Decode(AssetId id, NoSettings settings, ref SequenceReader reader) - { - return reader.ReadString(); - } - - public void HotSwap(string instance, string replacement) - { - throw new NotImplementedException(); - } - - public NoSettings ReadSettings(ref SequenceReader reader) - { - return default; - } - - public void WriteSettings(NoSettings settings, IBufferWriter writer) - { - - } -} diff --git a/source/CapriKit.Tests/AssetPipeline/RepeatTranscoder.cs b/source/CapriKit.Tests/AssetPipeline/RepeatTranscoder.cs deleted file mode 100644 index 549c482..0000000 --- a/source/CapriKit.Tests/AssetPipeline/RepeatTranscoder.cs +++ /dev/null @@ -1,49 +0,0 @@ -using CapriKit.AssetPipeline; -using CapriKit.IO; -using CapriKit.IO.Streams; -using System.Buffers; - -namespace CapriKit.Tests.AssetPipeline; - -internal readonly record struct RepeatSettings(int Count) : IAssetSettings -{ - public void Write(IBufferWriter writer) - { - writer.Write(Count); - } -} - -/// -/// Repeats the text in a text file times -/// -internal sealed class RepeatTranscoder : IAssetTranscoder -{ - public Guid Id => Guid.Parse("{0F1F51E7-2F2B-4E3B-9C93-15BBB61C1AF4}"); - public int Version => 1; - - public async Task Encode(AssetId id, RepeatSettings settings, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer) - { - var text = await fileSystem.ReadAllText(id.Path); - writer.Write(string.Concat(Enumerable.Repeat(text, settings.Count))); - } - - public string Decode(AssetId id, RepeatSettings settings, ref SequenceReader reader) - { - return reader.ReadString(); - } - - public void HotSwap(string instance, string replacement) - { - throw new NotImplementedException(); - } - - public RepeatSettings ReadSettings(ref SequenceReader reader) - { - return new RepeatSettings(reader.ReadInt32()); - } - - public void WriteSettings(RepeatSettings settings, IBufferWriter writer) - { - writer.Write(settings.Count); - } -} From 6f12104cfeb7ce8a73f8d44c40c8763cd5bb7a4e Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Sun, 9 Aug 2026 17:00:02 +0200 Subject: [PATCH 27/53] Add review --- Research/AssetPipelineRewriteReview.md | 726 +++++++++++++++++++++++++ 1 file changed, 726 insertions(+) create mode 100644 Research/AssetPipelineRewriteReview.md diff --git a/Research/AssetPipelineRewriteReview.md b/Research/AssetPipelineRewriteReview.md new file mode 100644 index 0000000..eb89218 --- /dev/null +++ b/Research/AssetPipelineRewriteReview.md @@ -0,0 +1,726 @@ +# Asset Pipeline Rewrite Review + +> Multi-agent review of the rewritten `source/CapriKit.AssetPipeline` and +> `source/CapriKit.AssetPipeline.DirectX11` (branch `feature/asset_pipeline`, commit `5a4a056` +> "Complete redo asset pipeline"). Four review agents ran in parallel — correctness, +> ease-of-use/DX/clarity, performance, and modern .NET constructs — each reading the full +> source plus its `CapriKit.IO` dependencies. Findings were deduplicated, re-ranked and +> cross-checked against a manual read; the blocking section was verified independently +> against the source before being written down. +> +> Supersedes `AssetPipelineReview.md`, which reviewed the *pre-rewrite* design. + +**Headline: the code cannot currently run end-to-end.** `AssetManager`'s constructor throws +for exactly the configuration `AddAssetPipeline` builds. That is consistent with the rest of +the evidence — commit `5a4a056` deleted all five `CapriKit.Tests/AssetPipeline/*` files without +replacement, and nothing in the repo calls `AssetManager.Load`. Read this document as a review +of a design that has not executed yet, not of working code. + +## Overview + +| # | Cluster | Finding | Severity | Location | +|---|---------|---------|----------|----------| +| B1 | Blocking | `AssetManager` ctor always throws — `Watch(DirectoryPath.Empty)` is rejected by every filesystem | Critical | `HotReloadManager.cs:38` | +| B2 | Blocking | Every shader with an `#include` fails to build — include root resolved against process CWD | Critical | `VertexShaderTranscoder.cs:14` | +| B3 | Blocking | Assets are never disposed — the `Line` wrapper is cast to `IDisposable`, not the asset | Critical | `AssetCache.cs:79,87` | +| B4 | Blocking | `AssetManager.Dispose` never disposes `HotReloadManager`, leaking the OS watcher | High | `AssetManager.cs:117` | +| C1 | Threading | Threading model undecided: async continuations mutate main-thread state | High | `AssetManager.cs:110`, `AssetCache.cs:62`, `HotReloadManager.cs:51` | +| C2 | Correctness | `#include`d files never hot-reload — dependency keys and watcher events use different path shapes | High | `VertexShaderTranscoder.cs:13-14` | +| C3 | Correctness | A failed encode destroys the last good build artifact | Medium | `AssetEncoder.cs:22` | +| C4 | Correctness | Concurrent `Load` of the same id throws | Medium | `AssetManager.cs:24` | +| C5 | Correctness | `IsUpToDate` never checks the primary source; `<` should be `!=` | Medium | `AssetManager.cs:93,102` | +| C6 | Correctness | Successful hot reload logged at `LogLevel.Error` | Low | `HotReloadManager.cs:161` | +| C7 | Correctness | Leftover `v2` namespace from the rewrite | Low | `HotReloadManager.cs:8` | +| D1 | DX | `Load` returns an `IDisposable` the caller must never dispose | High | `AssetManager.cs:24` | +| D2 | DX | The reader-lifetime rule is a `//` comment, so it never reaches IntelliSense | High | `IAssetTranscoder.cs:28-30` | +| D3 | DX | `AssetId(string Key, FilePath Path)` — swapping the arguments compiles | Medium | `Asset.cs:10` | +| D4 | DX | Build artifacts land beside sources; `.cka` not in `.gitignore` | Medium | `AssetUtilities.cs:11` | +| D5 | DX | Transcoder + settings passed on every single `Load` | Medium | `AssetManager.cs:24` | +| D6 | DX | Documentation gaps and one misleading `` | Medium | `IAssetTranscoder.cs:6-16` | +| D7 | DX | Error messages: one bare `Exception`, one raw `KeyNotFoundException`, one silent blackout | Medium | `AssetCache.cs:27,56`, `AssetDecoder.cs:78` | +| P1 | Perf | Every load reads the whole file twice | High | `AssetManager.cs:35,38` | +| P2 | Perf | `ArrayPool.Shared` silently stops pooling above 1 MB → LOH churn | High | `AssetDecoder.cs:24,60` | +| P3 | Perf | No `ConfigureAwait(false)` anywhere; a 50–500 ms `D3DCompile` can land on the main thread | Medium | 12 awaits across 4 files | +| P4 | Perf | First-run path throws an exception as normal control flow | Medium | `AssetDecoder.cs:55` | +| P5 | Perf | Redundant `stat` syscalls, per-load `Guid.Parse`, per-load transcoder allocation | Low | `AssetDecoder.cs:17,54` | +| M1 | Modern | Constrain `TSettings : IEquatable`; `NoSettings` as `readonly record struct` | High | `IAssetTranscoder.cs:17`, `AssetManager.cs:81-90` | +| M2 | Modern | No `CancellationToken` anywhere, unlike the rest of `CapriKit.IO` | Medium | `AssetManager.cs:24`, `IAssetTranscoder.cs:26` | +| M3 | Modern | Assorted one-line modernisations, no downside | Low | various | +| M4 | Modern | Library is AOT/trim-clean but unguarded — `IsAotCompatible` set nowhere | Low | `Directory.Build.props` | + +**Ground truth** (verified): `net10.0` (`net10.0-windows` for DirectX11), `LangVersion Latest` → C# 14, +`Nullable enable` with `WarningsAsErrors`, version `0.1.0-alpha`. No AOT/trim properties set anywhere +in the repo. Both projects compile with 0 warnings, so nothing below is flagged by the compiler. + +--- + +## Blocking — prevents the pipeline running at all + +### B1 · `AssetManager`'s constructor always throws + +`HotReloadManager.cs:38`: + +```csharp +Watcher = fileSystem.Watch(DirectoryPath.Empty); // Watch for all changed, usually fileSystem is a ScopedVirtualFileSystem +``` + +`DirectoryPath.Empty` is `new(null)` (`DirectoryPath.cs:12`), so `Path == ""` and +`IsAbsolute` is `Path.IsPathFullyQualified("")` → `false`. +`ReadOnlyScopedFileSystem.Watch` (`ScopedFileSystem.cs:143`) begins with +`ThrowIfPathIsOutsideBasePath(directory)`, which `Debug.Assert(path.IsAbsolute)` fires on in +Debug builds and then throws `ForbiddenPathException` because `""` does not start with the base +path. A plain `FileSystem` is no better: `Path.GetFullPath("")` throws `ArgumentException`. + +`AssetManager` constructs a `HotReloadManager` unconditionally (`AssetManager.cs:21`), and +`AddAssetPipeline` (`ServiceCollectionExtensions.cs:15`) supplies exactly a +`new FileSystem().ScopedTo(assetDirectory)`. So the documented entry point throws before you can +load anything. + +The intent is already stated in the comment on that line, and the overload exists: + +```csharp +Watcher = fileSystem.Watch(); // ScopedFileSystem.cs:151 — watches BasePath, no path check +``` + +It currently lives only on `ReadOnlyScopedFileSystem`, so it needs promoting to +`IReadOnlyVirtualFileSystem` to be reachable through the interface. The alternative is to make +`ThrowIfPathIsOutsideBasePath` treat an empty relative path as "the base path itself". + +### B2 · Every shader with an `#include` fails to build + +`VertexShaderTranscoder.cs:11-15`: + +```csharp +var source = await fileSystem.ReadAllText(id.Path); +var includePath = id.Path.Directory; // relative, e.g. "shaders/" +var bytes = ShaderCompiler.CompileVertexShader(fileSystem, includePath, source, id.Key, id.ToString()); +``` + +`includePath` is relative. `ShaderIncludeResolver`'s constructor wraps it in a +`ReadOnlyScopedFileSystem`, whose constructor does `BasePath = basePath.ToAbsolute()` +(`ScopedFileSystem.cs:49`) → `Path.GetFullPath("shaders/")`, resolved against +**`Environment.CurrentDirectory`**. + +With assets at `C:/game/content` and the process running from `C:/game/bin`, the resolver builds +`C:/game/bin/shaders/foo.hlsl` and hands it to the underlying scoped filesystem, which rejects it +with `ForbiddenPathException`. Shaders only compile when CWD happens to equal the asset root. + +Fix: resolve the include directory against the filesystem's base rather than the CWD — pass the +already-scoped filesystem plus the relative directory and let `ShaderIncludeResolver` keep paths +relative instead of calling `ToAbsolute()`. + +### B3 · Assets are never disposed + +`AssetCache.cs:76-80` and `:83-90`: + +```csharp +Lines.Remove(key, out var entry); +(entry as IDisposable)?.Dispose(); // entry is Line, not entry.asset +``` + +`Line` (`AssetCache.cs:11-15`) does not implement `IDisposable`, so the `as` cast is *always* null +and the `?.` is always a no-op. The same defect is in `Dispose()`, where `value` is also a `Line`. + +Load a shader, unload it, call `Update()`: the entry leaves the dictionary and the `VertexShader` +— along with its `ID3D11VertexShader` — is silently dropped. Every collected asset leaks its +native handle, `AssetManager.Dispose()` leaks all live ones, and D3D11 will report live-object +warnings on device release. + +```csharp +(entry?.asset as IDisposable)?.Dispose(); +(value.asset as IDisposable)?.Dispose(); +``` + +The naming inside `Line` is what hides this — PascalCase primary-constructor parameters shadowing +camelCase public fields, five lines apart. Renaming the type to `Entry` and the field to `Asset` +makes the bug visible at a glance. + +### B4 · `AssetManager.Dispose` never disposes `HotReloadManager` + +`AssetManager.cs:117-120` disposes only `Cache`. `HotReloadManager.Dispose` is the only caller of +`Watcher.Stop()` (`HotReloadManager.cs:148`), and grep confirms nothing in the repo invokes it. +After disposal the `FileSystemWatcher` keeps running, keeps enqueueing into +`FileSystemEventQueue`, and keeps the whole manager graph — including disposed D3D wrappers — +alive. Creating and disposing several `AssetManager`s leaks one OS watcher handle each. + +--- + +## Correctness + +### C1 · The threading model is undecided — and that is the root defect + +Three separate findings share one cause. `AssetManager.Load` is `async`; with no +`SynchronizationContext` (normal for a game loop) the continuation after +`await AssetDecoder.Decode(...)` runs on a thread-pool thread. `RegisterAsset` +(`AssetManager.cs:110-115`) then mutates main-thread state from there: + +- **`AssetCache.Collect`/`Dispose` take no lock** (`AssetCache.cs:62`, `:83`) while `Put`, `TryLease` + and `Return` all do. `Lines.Add` from a loader thread during `Collect`'s `foreach` gives + `InvalidOperationException: Collection was modified`, or a read racing a dictionary resize. +- **Zero-refcount resurrection.** `Collect` snapshots a `refCount == 0` entry into `toCollect` + (`:64-72`); before the removal loop at `:76` a background `TryLease` can bump it to 1 and hand the + asset to a caller. `Collect` then removes and (once B3 is fixed) disposes it. The caller holds a + disposed asset, and its later `Unload` throws `KeyNotFoundException` at `:56`. +- **`HotReloadManager` has no synchronisation at all.** `Track` (`:51-68`) writes the plain + `Dictionary` fields `Tracked` and `Dependents` while `DrainFileChanges` (`:91`) and `ReloadOne` + (`:114`) read them on the main thread. + +The class doc at `AssetCache.cs:6` also contradicts itself: "Assets can be leased and returned at +any time (though the class requires single-threaded access)" describes a class that nonetheless +takes a `Lock`. + +**Cheapest coherent fix:** have `RegisterAsset` enqueue onto a `ConcurrentQueue` that `Update()` +drains on the main thread, mirroring how `PendingReloads` already works. The cache then genuinely +*is* main-thread-only, the `Lock` can be deleted rather than extended, and the doc comment becomes +true. Decide this before touching anything else in `AssetCache` — it determines whether the lock +stays at all. + +### C2 · `#include`d files never hot-reload + +The transcoder reads its primary file straight off the spy +(`fileSystem.ReadAllText(id.Path)`), so the spy records a **relative** path. Includes go through +`new ReadOnlyScopedFileSystem(spy, includePath)`, whose `OpenRead` calls +`Source.OpenRead(GetFilePath(file))` — resolving to **absolute** before the spy ever sees it. +Observed in an agent's repro: + +``` +spy recorded: shaders/a.hlsl +spy recorded: C:/Users/.../capri_repro_assets/shaders/a.hlsl +``` + +Meanwhile `ScopedFileSystemEventListener.cs:22` normalises every watcher event to +`e.File.GetPathRelativeTo(BasePath)` — always relative. So `DrainFileChanges` +(`HotReloadManager.cs:91`) looks up a relative key in a `Dependents` map keyed absolutely, and +misses. + +Edit a top-level `.hlsl` and it hot-reloads; edit an `#include`d one and nothing happens, ever. +`IsUpToDate` still works for these files because `ScopedFileSystem.Exists` accepts both shapes — +which is precisely why this would be easy to ship unnoticed. + +Fix: normalise dependency keys to one representation. Simplest is to have +`VirtualFileSystemSpy` record `path.GetPathRelativeTo(basePath)`. + +### C3 · A failed encode destroys the last good build artifact + +`AssetEncoder.cs:22` opens the `.cka` with `FileMode.Create` (truncate) *before* +`encoder.Encode` runs. A shader typo during hot reload throws out of `WritePayload` and leaves a +0-byte `.cka`. It is recovered on the next run — `TryDecodeBuildMetaData`'s blanket `catch` +(`AssetDecoder.cs:78`) returns null and forces a rebuild — so this is not silent corruption, but +the previously-good artifact is gone. Encode into a buffer first and open the output file only +once encoding succeeded. This is also a prerequisite for cancellation being safe (see M2). + +### C4 · Concurrent `Load` of the same `AssetId` throws + +There is no in-flight de-duplication. Two overlapping calls both miss `TryLease`, both call +`AssetEncoder.Encode` on the same output path (the second `CreateReadWrite` throws `IOException`), +and if they get past that, the second `Cache.Put` throws +`new Exception($"Cache already contains asset: {id}.")` (`AssetCache.cs:27`). The same collision +exists between a `Load` and a concurrent `HotReloadable.Reload` of the same asset. +Fix: a `Dictionary>` of in-flight loads. + +### C5 · `IsUpToDate` never checks the primary source, and compares timestamps with `<` + +`AssetManager.cs:93-105` iterates only `build.Dependencies`, which is whatever the transcoder +happened to open *through the spy*. `AssetEncoder.cs:19`'s `ThrowOnFileNotFound(id.Path, fileSystem)` +uses the raw filesystem, and `Exists` is not spied, so nothing guarantees `id.Path` appears in the +list. A transcoder that generates content or reads its source by another route produces an empty +dependency list — the `foreach` body never runs, `IsUpToDate` returns `true` unconditionally, and +the stale artifact is used forever. Always check `id.Path` explicitly, and/or seed the spy with it. + +Separately, `:102` uses `if (version < lastWrite)`. Restoring an older copy of a source file with +its mtime preserved (backup restore, `robocopy`, some VCS tooling) leaves `lastWrite <= version`, +so no rebuild happens and the wrong asset is used. `version != lastWrite` is the standard +formulation. + +### C6 · Successful reload logged at `Error` + +`HotReloadManager.cs:161` — `[LoggerMessage(Level = LogLevel.Error, Message = "Reloading asset completed: {asset}")]`, +copy-pasted from the adjacent `LogReloadFailed`. Every successful hot reload is reported as an +error to the log sink and to anything filtering on error level. + +### C7 · Leftover `v2` namespace + +`HotReloadManager.cs:8` still declares `namespace CapriKit.AssetPipeline.v2;` while every other +file moved to `CapriKit.AssetPipeline`; `AssetManager.cs:1`'s `using CapriKit.AssetPipeline.v2;` +exists purely to compensate. `git show --stat 5a4a056` shows `v2/` → `` renames for `AssetCache` +and `HotReloadable` but not for this file. A `v2` sub-namespace is visible to anyone typing +`CapriKit.AssetPipeline.` in IntelliSense. + +### Checked and found clean + +Worth recording, so these are not re-reviewed later: + +- **Encode/decode round trip.** Every field traced. `AssetEncoder` writes + `Guid, int | int len, settings | int len, payload | int count, (long ticks, string path)*` and + `AssetDecoder` reads exactly that, in that order, matching the format comment at + `AssetEncoder.cs:9`. Endianness matches (`BinaryPrimitives` LE on both sides), `Guid` uses + `bigEndian: false` both ways, `Write(FilePath)` binds to the `string` overload via the implicit + conversion and pairs correctly with `ReadString`'s 7-bit-length prefix, and `SliceUnread` + correctly advances the outer reader past each length-delimited section. No mismatch found. +- `build != default` (`AssetManager.cs:36`) does behave as a null check — the record-synthesized + `op_Inequality` handles null correctly. (Still worth changing for clarity; see M3.) +- The watcher **is** debounced (`HotReloadManager.cs:78-82`, 0.5 s of quiet), and the `.cka` is + written on the raw filesystem *before* the spy is created (`AssetEncoder.cs:22` vs `:24`), so + build artifacts are not recorded as their own dependencies. No infinite rebuild loop. +- `IVertexShader.HotSwap` (`VertexShader.cs:12-19`) correctly preserves the live object's identity + and disposes only the orphaned old `ID3D11VertexShader`, so existing holders stay valid. +- `AssetDecoder`'s `ArrayPool` rent/return is balanced in `finally` and the sequence is bounded to + the real length rather than the rented length. + +--- + +## Ease of use, DX and clarity + +### D1 · `Load` hands back an `IDisposable` the caller must never dispose + +`AssetManager.cs:24` returns a bare `TAsset`. For the only worked example that is `IVertexShader`, +which is `IDisposable`. The C# reflex on an `IDisposable` returned from a method is `using` — +which destroys an asset the cache still hands to every other caller, while the refcount never +notices. `Unload(id)` is the real release, and nothing in the type system, the name, or the docs +says so: `Load` and `Unload` have no XML documentation at all. + +The failure is symmetric. Over-unloading is silent — `AssetCache.Return` (`:52-59`) decrements past +zero without complaint and `Collect` then frees a live asset. Forgetting to unload leaks with no +diagnostic. + +**Small fix** — document the ownership contract and make `Return` refuse to go negative: + +```csharp +/// +/// Loads an asset, building it first if there is no up-to-date build artifact. +/// The returned instance is owned by the AssetManager and shared with every other +/// caller of the same : never dispose it, and never keep it past the +/// matching . Each successful Load takes one reference; call +/// exactly once per Load. +/// +``` + +```csharp +// AssetCache.Return +if (!Lines.TryGetValue(id, out var entry)) + throw new InvalidOperationException($"Cannot unload asset {id}: it was never loaded, or it was already unloaded once per Load."); +if (--entry.refCount < 0) + throw new InvalidOperationException($"Asset {id} was unloaded more often than it was loaded."); +``` + +**Medium fix**, matching the `AssetRef` idea in `AssetPipelineArchitecture.md`: return an +`AssetLease : IDisposable` with a `.Value`, so `using` becomes the *correct* thing rather +than the destructive one. Costs one struct plus `.Value` at every use site. + +### D2 · The most dangerous rule in the contract is in a `//` comment + +`IAssetTranscoder.cs:28-30` states that the reader's buffer is only valid for the duration of the +call and decoders must copy out anything they keep. That is true — `AssetDecoder.cs:24,43` rents +from `ArrayPool` and returns it in a `finally` — and it is invisible to a package consumer, +because `//` comments do not ship in the XML documentation file or IntelliSense. A decoder that +retains a `ReadOnlySequence` slice compiles fine and later reads recycled pool memory under load. + +Promote it to `` on `Decode`, along with the async rationale at `:20`: + +```csharp +/// +/// The reader is backed by a pooled buffer that is recycled the moment this method returns: +/// copy out (ToArray, CopyTo) anything you keep. Storing the reader, a slice of +/// it, or a span into it will silently read another asset's bytes later. +/// Runs synchronously, possibly on a worker thread; do not touch main-thread-only state. +/// +``` + +### D3 · `AssetId(string Key, FilePath Path)` — swapping the arguments compiles + +`Asset.cs:10`. `FilePath` has `implicit operator FilePath(string?)` (`FilePath.cs:104`), so +`new AssetId("shaders/basic.hlsl", "VsMain")` — path first, the order every reader will guess — +compiles cleanly and fails at runtime with `FileNotFoundException: ... VsMain`. There is also no +way to express "no sub-resource" other than `new AssetId("", path)`. + +```csharp +/// Virtual path of the source file the asset is built from. +/// Names a sub-resource inside — e.g. the HLSL entry +/// point for a shader. Empty means "the whole file". Assets with the same Path but different +/// Keys are separate assets with separate build artifacts. +public sealed record AssetId(FilePath Path, string Key = ""); +``` + +There are zero call sites today, so this is free now and will not be later. + +### D4 · Build artifacts land beside the sources + +`AssetUtilities.cs:7-16` maps `grass.png` → `grass.png.cka` **in the same directory**, and `.cka` +is not in `.gitignore`. `AssetPipelineArchitecture.md` specifies a mirrored compiled tree, +deliberately separate. First-run experience is that the user's art folder fills with `.cka` files +and their next `git status` is noise. + +Small: add `*.cka` to `.gitignore` and hoist the extension to a documented +`public const string BuildArtifactExtension`. Real fix: a `DirectoryPath outputDirectory` on the +`AssetManager` constructor and `AddAssetPipeline`, with `ToEncodedFilePath` rebasing onto it +(~15 lines, and it is the design already written down). + +### D5 · Transcoder and settings on every `Load` + +`AssetManager.cs:24` requires two extra arguments per call, both constant for the lifetime of the +program. `AssetManagerExtensions.cs:9-13` shows what people reach for and why it is not enough: + +```csharp +public static Task LoadVertexShader(this AssetManager assetManager, Device device, AssetId id) +{ + var transcoder = new VertexShaderTranscoder(device); // new instance per call + return assetManager.Load(id, transcoder, default); +} +``` + +A fresh transcoder per call means `HotReloadManager.Track` stores a distinct transcoder object per +asset, and `device` is threaded through every call site forever. + +Recommended small fix — bind transcoder and settings once, keep full compile-time typing, add no +registry and no "missing transcoder" runtime failure: + +```csharp +/// How to build and load one kind of asset. Create once, reuse for every Load. +public sealed record AssetSource(IAssetTranscoder Transcoder, TSettings Settings) + where TAsset : class; + +public Task Load(AssetId id, AssetSource source) where TAsset : class + => Load(id, source.Transcoder, source.Settings); +``` + +Alternatives and their cost: *registration* (`assets.Register(...)` + `Load(id)`) +reads best at the call site but reintroduces the `TranscoderCollection` deleted in `5a4a056`, +trades a compile error for a runtime "no transcoder registered", and forces settings to be +per-type rather than per-load. *DI* is worse here, because the transcoder needs the `Device` and +the container would have to know about graphics. A typed `AssetHandle` bundling id + transcoder ++ settings is the nicest end state but wants a per-`Device` catalog object. + +### D6 · Documentation + +Measured against the standard in `CLAUDE.md` ("teach the library user how to use this correctly +and when not to"). + +**Undocumented public members:** `AssetManager` itself — the entry point of the package — plus its +constructor, `Load`, `Unload` and `Dispose`; `IAssetTranscoder.Id` and `.Version`; `NoSettings`; +all four abstract members of `NoSettingsTranscoder`; `AddAssetPipeline`; +`AssetManagerExtensions.LoadVertexShader`. + +`Id`/`Version` is the highest-value gap in the file: they are the cache-invalidation keys, and +nothing tells an implementer that `Version` must be bumped whenever `Encode`'s output format +changes — otherwise every user's stale artifacts are fed to the new decoder. + +**Misleading docs:** + +- `IAssetTranscoder.cs:6-9` — "Interface for classes that builds assets ... and load them when the + program needs them" describes `AssetManager`, not a transcoder. `:16`'s + `` then copies that wrong sentence onto the generic + interface people actually implement. Better: "One transcoder owns both halves of a single asset + type: the offline build (`Encode`, dev machine) and the runtime load (`Decode`, game process)." +- `:33` — "Decodes the file created the `Encode`" is missing a word. +- `:39-50` — `WriteSettings` and `ReadSettings` have near-identical text, and `ReadSettings` says + it "Decodes the settings ... **into** the stream" when it reads *from*. Neither mentions that + `WriteSettings` doubles as the staleness comparison (see M1). +- `AssetCache.cs:6` — claims single-threaded access while the class locks. The truth is narrower + and more useful: lease/return are thread-safe; `Collect` and `Dispose` are main-thread only. +- `AssetManager.cs:64` — `Update` has a `` but no ``, so IntelliSense shows an + empty description with a floating remark. It never says what `Update` does or what breaks if you + skip it (nothing is ever freed; hot reload silently does nothing). Consider a + `[Conditional("DEBUG")]` main-thread assertion so "must" is enforced rather than hoped for. + +### D7 · Failure modes surfaced to the user + +| Situation | Today | Verdict | +|---|---|---| +| Source file missing | `FileNotFoundException` naming the path — `AssetManager.cs:48` | Good | +| Build artifact missing at decode | `FileNotFoundException` naming file and asset — `AssetDecoder.cs:19` | Good | +| Artifact built by another transcoder | `InvalidDataException` naming both GUIDs and versions — `AssetDecoder.cs:132` | Good | +| `ThrowOnFileNotFound` in the encoder | `new FileNotFoundException(null, path)` — `AssetUtilities.cs:22` | Bad: `null` message, no asset context | +| Same id loaded twice concurrently | `throw new Exception(...)` — `AssetCache.cs:27` | Bad: uncatchable by type | +| `Unload` of an id never loaded | `KeyNotFoundException` — `AssetCache.cs:56` | Worst: names neither the asset nor the API | +| Same id loaded as two different types | `InvalidCastException`, no id — `AssetCache.cs:44` | Bad | +| **Corrupt build artifact** | `catch { return null; }` — `AssetDecoder.cs:78` | Worst for diagnosis | + +That last row is the answer to "what does it say when an artifact is corrupt": *nothing*. It +silently rebuilds, which is the right recovery, but a permission error, a >2 GB +`OverflowException` (`AssetDecoder.cs:59`) and a genuine bug in someone's `ReadSettings` all look +identical and produce a rebuild on **every single load**, forever, with zero log output — +`TryDecodeBuildMetaData` has no `ILogger`. Pass the logger in and log once at Debug/Warning. + +### Naming + +- `AssetCache.Line` — "line" is a CPU-cache term; this is an entry, and the local variable is + literally called `entry`. Renaming to `Entry` (and the field to `Asset`) is what exposes B3. +- `Asset` — a type named `Asset`, with a type parameter named `TAsset`, and a + member `Value` of that type. `LoadedAsset` reads better; it is internal, so this is cheap. +- `HotSwap(TAsset instance, TAsset newParts)` — `newParts` is not parts, it is a fully constructed + replacement. `HotSwap(TAsset live, TAsset replacement)`, and the doc should state who disposes + the leftover wrapper: `VertexShader.cs:12-19` disposes the old `ID3D11VertexShader` but not the + `newParts` wrapper, and that asymmetry needs saying. +- **"Transcoder" already means something else in this repo.** `CapriKit.SuperCompressed` has + `Ktx2Transcoder`, using *transcode* in its Basis-Universal sense: compressed container → GPU + format, at runtime. A future `TextureTranscoder : IAssetTranscoder` + will call `Ktx2Transcoder.Transcode(...)` inside its `Decode` — two unrelated meanings of the + same word in one file. `AssetPipelineArchitecture.md` names these **Compilers** and **Loaders**. + The one-interface-does-both design is genuinely better than MonoGame's four-piece split and is + worth keeping; only the name collides. Suggest taking the doc fix now and revisiting the name + when the texture transcoder lands and the collision becomes concrete. + +### Ceremony cost in DirectX11 — proportionate + +| File | Lines | Real statements | +|---|---|---| +| `VertexShaderTranscoder.cs` | 29 | 6 | +| `ShaderTranscoder.cs` | 26 | 8 | +| `AssetManagerExtensions.cs` | 14 | 2 | +| **Total per asset type** | **69** | **16** | + +`NoSettingsTranscoder` is doing its job — roughly 4:1 lines-to-logic for a binary-serialized GPU +resource is fine, and there is no per-type "content writer + content reader + settings class" tax +like MonoGame's. The only remaining pure forwarding is `HotSwap` (one line delegating to the +asset's own `HotSwap`); an optional `HotSwappableTranscoder` base would remove 4 lines per +transcoder but is not worth it until there are three or more. + +--- + +## Performance + +Judged against two standards: the **encode** path runs rarely and offline-ish, so allocation +matters little; the **load** path runs during gameplay or level load, where allocations, copies +and main-thread stalls are expensive. `Update()` runs on the main thread every frame. + +### P1 · Every load reads the whole file twice + +`AssetManager.cs:35` calls `TryDecodeBuildMetaData`, which `ReadExactlyAsync`s the *entire* file — +payload included — just to `SkipPayload` (`AssetDecoder.cs:69`) and reach the dependency list at +the tail. Then `:38` calls `Decode`, which re-opens and re-reads the identical bytes. + +Cost: 2× file bytes, 2× `FileStream` open, 2× buffer rent, per asset, on the common path. A 4 MB +texture costs 8 MB of IO. File open on Windows is ~50–500 µs cold, so the duplicate opens alone +are ~50–500 ms per 1000 assets. + +Fix: one pass. Read the file once into a pooled buffer, parse header → settings → *retain the +payload slice* → skip to dependencies → run the up-to-date check → and only then call +`transcoder.Decode` on the retained slice. `SliceUnread` already gives exactly the slice needed; +the file format does not change. **Biggest single item in this review.** + +### P2 · `ArrayPool.Shared` silently stops pooling above 1 MB + +`AssetDecoder.cs:24,60`. `ArrayPool.Shared` caps at `MaxArrayLength = 1024*1024`; above that +`Rent` calls `GC.AllocateUninitializedArray` and `Return` drops the array on the floor. So for any +asset over 1 MB — every texture and mesh — this is a plain **LOH allocation of the full file size, +twice per load** (see P1), each becoming garbage immediately. A 16 MB texture is 32 MB of LOH +garbage per load; 100 such assets is 3.2 GB of churn and forced gen2/LOH pressure during level +load, exactly the main-thread stall case that matters. + +Fix: a dedicated `ArrayPool.Create(maxArrayLength: 64MB, maxArraysPerBucket: 2)` held static +on `AssetDecoder`, or a single reusable staging buffer on `AssetManager` (loads are already +serialised through the same manager). Fixing P1 first halves this for free. + +### P3 · No `ConfigureAwait(false)` anywhere + +12 awaits across `AssetManager.cs`, `AssetDecoder.cs`, `AssetEncoder.cs`, `HotReloadable.cs`. +`CapriKit.IO` uses it on all 6 of its awaits, so the pipeline is inconsistent with its own +dependency. If a host installs a `SynchronizationContext` — an editor or tooling host, an +ImGui-driven tool, a test harness — every continuation posts back to the main thread. The worst +case is concrete: `VertexShaderTranscoder.cs:13-15` awaits `ReadAllText` and then runs a +synchronous `D3DCompile` of **50–500 ms** on the continuation. + +Add `.ConfigureAwait(false)` throughout and enable CA2007 so it stays enforced. Note this makes +C1's race explicit rather than worse — settle the threading model first, and do not treat adding +it as closing the threading question. + +Related: `Load` runs synchronously on the caller's thread up to the first real yield, including +the blocking `FileStream` open. Either document that `Load` must be called off the main thread, or +open the file on a worker. + +### P4 · The first-run path throws as normal control flow + +`AssetDecoder.cs:55` throws `FileNotFoundException` when the artifact does not exist yet, caught by +the blanket handler at `:78`. "Asset has never been built" is the expected first-run state, not an +error. Undebugged that is ~5–50 µs; **under a debugger, first-chance exception notifications cost +1–10 ms each**, so a fresh build of 1000 assets becomes 1–10 seconds of pure debugger stall and +floods the exception window. One line: `if (!fileSystem.Exists(inputPath)) { return null; }` before +the `try`. Cheapest high-value fix in this section. + +### P5 · Smaller items + +- **~6 redundant `stat` syscalls per load.** `AssetDecoder.cs:17,54` call `Exists`, then + `OpenRead` → `FindOrThrow` → `GetFileInfo().Exists`, then the actual open — and all of it twice + because of P1. `FileMode.Open` already throws `FileNotFoundException`, so `FindOrThrow` is pure + duplication. ~30–90 ms per 1000-asset level load; the fix is *fewer* lines of code. +- **`Guid.Parse` on every transcoder construction** (`VertexShaderTranscoder.cs:9`) combined with a + new transcoder per `Load` (`AssetManagerExtensions.cs:11`). `static readonly Guid` and a cached + per-`Device` transcoder. +- **`IsUpToDate`'s two `ArrayBufferWriter`s** are ~64 bytes per load for `NoSettings` and ~600 + bytes for a realistic settings struct — real but noise next to P1/P2. The *structural* waste + matters more: `build.Settings` was just deserialised purely so it could be re-serialised and + compared byte-for-byte. See M1. +- **`ArrayBufferWriter` growth in `AssetEncoder`** (`:53`) doubles and discards, never using + `ArrayPool`; combined with `PipeWriter` buffering everything until `FlushAsync` at `:31`, peak + memory is ~3× payload size. Normally an encode-path shrug, except this runs during hot reload + while the game is live, so the LOH churn causes gen2 pauses mid-play. Dropping `PipeWriter` for a + single size-hinted `ArrayBufferWriter` + `output.WriteAsync` is simpler *and* faster. + +### Explicitly judged not worth it + +- **`Collect()`'s O(n) scan.** Genuinely zero-allocation in steady state (struct enumerator; + `toCollect` only allocates when there is something to collect). ~5–20 µs/frame at 1000 assets + ≈ 0.1% of a 16.6 ms budget. Restructuring to a candidate list pushed from `Return()` is worth + doing for *clarity*, and the O() improvement is a bonus — not the reason. +- **`ValueTask` for `Load`.** ~72 bytes per load against a public signature change and + `.AsTask()` at any call site that stores the result. If synchronous cache hits matter, the honest + fix is a genuinely synchronous `TryGetLoaded(id, out asset)`, which needs no `ValueTask`. +- **Lock contention in `AssetCache`.** Uncontended `Lock` is ~20 ns and `Load` is not per-frame. + Do not remove the lock for speed (C1 may remove it for a better reason). +- **`Stopwatch.GetElapsedTime` every frame** (`HotReloadManager.cs:78`) — ~20–25 ns. Negligible. + +### Benchmarks + +`CapriKit.Benchmarks` uses `BenchmarkSwitcher.FromAssembly`, so a `[MemoryDiagnoser]` class drops +straight in — but it currently references only `CapriKit.Concurrency` and `CapriKit.IO`, so a +project reference is needed. Worth measuring, against `InMemoryFileSystem` to isolate CPU from +disk: `Load` on the cold-but-up-to-date path (the direct before/after for P1 and P5), +`AssetCache.Collect()` vs entry count at 100/1,000/10,000 (the one estimate above derived from +first principles rather than measurement), and `AssetEncoder.Encode` peak allocation across +payload sizes. **Fix B1 first** — none of this is currently exercisable end-to-end. + +--- + +## Modern .NET + +Targeting `net10.0` / C# 14. House style already in use elsewhere in the repo, so these are safe: +`SearchValues` (`IOUtilities.cs:11-12`), `ValueTask` and optional `CancellationToken` on every +async IO helper (`VirtualFileSystemExtensions.cs:18-63`), `readonly record struct`, primary +constructors, collection expressions, `[LoggerMessage]`. Never used anywhere in the repo: +static abstract interface members, `FrozenDictionary`, `TimeProvider`, `CollectionsMarshal`, +generic math. `System.Threading.Lock` appears exactly once — in `AssetCache.cs:17`, so the code +under review is already the most modern locking in the repo. + +### M1 · `where TSettings : IEquatable` + `readonly record struct NoSettings` + +`AssetManager.cs:81-90` decides staleness by serializing *both* settings objects into fresh +`ArrayBufferWriter`s and comparing spans. Nothing in `IAssetTranscoder.cs:39-43` tells the +implementer that their serializer must be **deterministic** — iterate a `Dictionary`, write a +`float` through a culture-sensitive path, or include a timestamp, and every `Load` decides the +artifact is stale and rebuilds forever, with no error. + +**The agents disagreed here, and the disagreement is instructive.** One proposed +`EqualityComparer.Default.Equals(...)`; another argued *against* it, because it silently +degrades to reference equality for a settings type that forgets to implement equality — a +"rebuild forever" cliff that fails silently, i.e. exactly the bug it was meant to remove. Adding +the constraint answers that objection directly: + +```csharp +public interface IAssetTranscoder + where TAsset : class + where TSettings : IEquatable + +// AssetManager.IsUpToDate +if (!settings.Equals(build.Settings)) { return false; } +``` + +and `public readonly record struct NoSettings;` gets `IEquatable`, `Equals`, +`GetHashCode` and `==` from one word. Nine lines become one, two allocations and two serializations +leave every load, and the hidden determinism contract disappears rather than needing to be +documented. Source-breaking — which at `0.1.0-alpha`, with `NoSettings` as the only settings type +in the repo, is the right moment. + +### M2 · `CancellationToken` — a real gap, with a caveat + +No method in the pipeline takes a token, though `CapriKit.IO`'s async helpers all accept one and +`AssetDecoder.cs:28,64` calls `ReadExactlyAsync` without one. Add +`CancellationToken token = default` as the trailing parameter and thread it into +`IAssetTranscoder.Encode`, which is where the seconds actually go. + +Honest caveat: this buys nothing on its own. `ShaderCompiler.CompileVertexShader` is a synchronous +blocking call inside an `async` method, so a token only takes effect at the cheap IO awaits — and +cancelling mid-`Encode` leaves a truncated `.cka` (C3). Add the parameter, but treat the +"cancel the loading screen" story as unfinished until transcoders poll and the write is atomic. +Source-breaking for `IAssetTranscoder.Encode`, not for `AssetManager.Load`. + +### M3 · One-line modernisations with no downside + +- `build is not null` instead of `build != default` (`AssetManager.cs:36`) — `!= default` on a + record dispatches through the synthesized `op_Inequality` instead of a plain null test. +- `catch (Exception ex) when (ex is not OperationCanceledException)` at `AssetDecoder.cs:78` — the + bare `catch` today also eats `OutOfMemoryException`, and once M2 lands it would convert "the user + cancelled" into "the metadata was unreadable, rebuild from scratch". +- `toCollect ??= []` (`AssetCache.cs:69`). +- Seal the records (`Asset.cs:10,12,15`) — `AssetId` is a `Dictionary` key in three places, so + sealing removes the `EqualityContract` virtual call from every lookup and makes the classic + derived-record-never-equals-base cache miss impossible. `record class` is also a redundant + spelling of `record`. +- `[LoggerMessage]`: drop `static` and the `ILogger` parameter (the generator resolves the field on + the containing type; both classes have one), removing a repeated argument from 13 call sites. + PascalCase the placeholders per CA1727. And `AssetManager.cs:122-128` logs *every* asset load at + `Information` — `Debug` fits a level load of a few thousand assets better. +- `static readonly Guid` instead of `Guid.Parse` per construction + (`VertexShaderTranscoder.cs:9`). +- `PendingRebuilds.First()` (`HotReloadManager.cs:107`) boxes `HashSet.Enumerator` through + `IEnumerable`. A `Queue` alongside the `HashSet` gives dedup *and* deterministic + reload order instead of hash order. +- `ObjectDisposedException.ThrowIf(disposed, this)` on `Load`/`Update`/`Put`/`TryLease` — house + style already in `CapriKit.SuperCompressed`. Today, using an `AssetManager` after `Dispose` + silently succeeds against an emptied cache. +- Drop the redundant `public` on `IAssetTranscoder.cs:26,36,43,50` (`:59` already omits it). +- `FileChances` → `FileChanges` (`HotReloadManager.cs:22,39,89`). + +### M4 · AOT/trim: clean today, unguarded tomorrow + +No reflection, no `Activator.CreateInstance`, no `MakeGenericType`, no `dynamic`, no generic +*virtual* methods; `ServiceCollectionExtensions.cs:12` uses the lambda-factory `AddSingleton` +overload rather than reflection-based activation. **The library is AOT/trim-clean as written.** +`true` is set nowhere in the repo — adding it turns on the +analysers so the property stays true when a future transcoder reaches for reflection. One line, no +code change, and it is the kind of guarantee a game library gets asked for. + +### Considered and rejected + +- **`System.IO.Hashing` / `XxHash3` for freshness.** Content hashing means *reading every + dependency file* on every load — strictly more IO on the path you most want fast. The one place + it would pay is hot reload: an editor that rewrites a file unchanged currently triggers a full + rebuild, and hashing only the file that raised the event would skip that. Worth doing *if* + spurious rebuilds annoy you in practice. Note `System.IO.Hashing` is declared at + `Directory.Packages.props:20` but referenced by no project — a dead central-package entry. +- **`FrozenDictionary`.** Every map here is write-often (`Lines`, `Tracked`, `Dependents`). Frozen + collections are for build-once-read-forever. No fit. +- **Static abstract interface members for `Id`/`Version`.** Would make version drift a + compile-time fact, but transcoders are *instances* carrying state (`VertexShaderTranscoder` holds + a `Device`), so every method in `AssetEncoder`/`AssetDecoder`/`AssetManager` would need an extra + `TTranscoder` type parameter — four files churned for harder-to-read generic signatures. +- **`TimeProvider`** for the hot-reload debounce (`HotReloadManager.cs:17,47,78`) — a 1:1 API swap + that would make the 0.5 s `MinWaitTime` testable without `Thread.Sleep`. Take it only if you + intend to test the debounce; it needs `Microsoft.Extensions.TimeProvider.Testing` added to + `Directory.Packages.props` and the repo uses `TimeProvider` nowhere. +- **`MemoryPool` + `using`** instead of the `ArrayPool` `try/finally` — `SequenceReaders.Create` + takes a `byte[]`, so you would need `MemoryMarshal.TryGetArray` to get back out. Worse than what + it replaces. +- **`IAsyncEnumerable`, `field`, `required`/`init`, generic math.** No batch-load API to convert, + no hand-written backing fields, no object-initializer surface, no numeric-generic code. + +--- + +## Tests + +Commit `5a4a056` deleted `AssetDecoderTests.cs`, `AssetEncoderTests.cs`, `AssetManagerTests.cs`, +`DummyTranscoder.cs` and `RepeatTranscoder.cs`, and nothing replaced them — there is no +`AssetPipeline` folder under `CapriKit.Tests` at all, though `CapriKit.Tests.csproj` still +project-references `CapriKit.AssetPipeline` and `Directory.Build.props:103` already grants +`InternalsVisibleTo`. + +Per `source/CapriKit.Tests/README.md` the bar is the happy path on anything with some complexity. +A round-trip test — encode → decode → assert equal, plus a second `Load` that hits the cache — +against `InMemoryFileSystem` would be roughly 40 lines and would have caught B3 immediately. + +`CapriKit.AssetPipeline` is also one of only two `source/` projects without a `README.md`, and it +is the one whose file format (`AssetEncoder.cs:9`) and hot-reload threading model most need a page +of prose. + +--- + +## Suggested order + +1. **B1 and B2** — nothing runs without them. +2. **Write back the happy-path round-trip test.** It is the thing that turns the rest of this list + from review comments into a safety net. +3. **B3, B4** — small, mechanical, and each one is a resource leak. +4. **Decide the threading model (C1)** before touching `AssetCache` further; it determines whether + the `Lock` stays at all, and P3 depends on the answer. +5. **D3, D4, M1** while there are still zero call sites and the breaking changes are free. +6. **P1 and P4** — the two largest performance wins, and P1 subsumes several smaller items. +7. The documentation pass (D1, D2, D6) and the error-message cleanup (D7). From a451213a75c0ecd8459e44ee1f6d16863173697d Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Sun, 9 Aug 2026 20:36:37 +0200 Subject: [PATCH 28/53] WIP so many bugs --- source/CapriKit.AssetPipeline/AssetCache.cs | 39 ++++++++++--------- source/CapriKit.AssetPipeline/AssetManager.cs | 5 +-- .../HotReloadManager.cs | 8 ++-- .../Shaders/ShaderIncludeResolver.cs | 11 ++++-- source/CapriKit.IO/DirectoryPath.cs | 11 ------ source/CapriKit.IO/FilePath.cs | 2 - source/CapriKit.IO/FileSystem.cs | 8 +++- source/CapriKit.IO/IVirtualFileSystem.cs | 12 ++++++ source/CapriKit.IO/InMemoryFileSystem.cs | 4 ++ .../ReadOnlyVirtualFileSystemSpy.cs | 4 ++ source/CapriKit.IO/ScopedFileSystem.cs | 15 +++++-- source/CapriKit.IO/VirtualFileSystemSpy.cs | 4 ++ .../AssetPipeline/AssetManagerTests.cs | 10 +++++ .../AssetPipeline/HotReloadManagerTests.cs | 15 +++++++ source/CapriKit.Tests/CapriKit.Tests.csproj | 1 + .../IO/ScopedFileSystemTests.cs | 14 +++++++ 16 files changed, 115 insertions(+), 48 deletions(-) create mode 100644 source/CapriKit.Tests/AssetPipeline/AssetManagerTests.cs create mode 100644 source/CapriKit.Tests/AssetPipeline/HotReloadManagerTests.cs diff --git a/source/CapriKit.AssetPipeline/AssetCache.cs b/source/CapriKit.AssetPipeline/AssetCache.cs index eff1e42..de9ebc5 100644 --- a/source/CapriKit.AssetPipeline/AssetCache.cs +++ b/source/CapriKit.AssetPipeline/AssetCache.cs @@ -3,15 +3,15 @@ namespace CapriKit.AssetPipeline; /// -/// Simple cache that uses reference counting to decide when to clean-up a resource. Assets can be leased and returned at any time (though the class requires single-threaded access). +/// Simple cache that uses reference counting to decide when to clean-up a resource. Assets can be leased and returned at any time. /// The actual disposing of objects only happens when the main thread calls . /// internal sealed class AssetCache : IDisposable { - private class Line(object Asset, int RefCount) + private class Line(object asset, int refCount) { - public readonly object asset = Asset; - public int refCount = RefCount; + public object Asset { get; } = asset; + public int RefCount { get; set; } = refCount; } private readonly Lock Lock = new(); @@ -39,8 +39,8 @@ public bool TryLease(AssetId id, [NotNullWhen(true)] out TAsset? asset) { if (Lines.TryGetValue(id, out var entry)) { - entry.refCount = entry.refCount + 1; - asset = (TAsset)entry.asset; + entry.RefCount = entry.RefCount + 1; + asset = (TAsset)entry.Asset; return true; } } @@ -54,7 +54,7 @@ public void Return(AssetId id) lock (Lock) { var entry = Lines[id]; - entry.refCount = entry.refCount - 1; + entry.RefCount = entry.RefCount - 1; } } @@ -62,21 +62,24 @@ public void Return(AssetId id) public void Collect() { List? toCollect = null; - foreach (var (key, value) in Lines) + lock (Lock) { - if (value.refCount <= 0) + foreach (var (key, value) in Lines) { - toCollect = toCollect ?? []; - toCollect.Add(key); + if (value.RefCount <= 0) + { + toCollect = toCollect ?? []; + toCollect.Add(key); + } } - } - if (toCollect == null) { return; } + if (toCollect == null) { return; } - foreach (var key in toCollect) - { - Lines.Remove(key, out var entry); - (entry as IDisposable)?.Dispose(); + foreach (var key in toCollect) + { + Lines.Remove(key, out var entry); + (entry?.Asset as IDisposable)?.Dispose(); + } } } @@ -84,7 +87,7 @@ public void Dispose() { foreach (var value in Lines.Values) { - (value as IDisposable)?.Dispose(); + (value.Asset as IDisposable)?.Dispose(); } Lines.Clear(); } diff --git a/source/CapriKit.AssetPipeline/AssetManager.cs b/source/CapriKit.AssetPipeline/AssetManager.cs index 28fc005..a999a9f 100644 --- a/source/CapriKit.AssetPipeline/AssetManager.cs +++ b/source/CapriKit.AssetPipeline/AssetManager.cs @@ -12,8 +12,7 @@ public sealed partial class AssetManager : IDisposable private readonly AssetCache Cache; private readonly HotReloadManager HotReloadManager; - - public AssetManager(ILoggerFactory logger, IVirtualFileSystem fileSystem) + public AssetManager(ILoggerFactory logger, ScopedFileSystem fileSystem) { Logger = logger.CreateLogger(); FileSystem = fileSystem; @@ -98,7 +97,7 @@ private bool IsUpToDate(IAssetTranscoder t } var lastWrite = FileSystem.LastWriteTime(file); - if (version < lastWrite) + if (version != lastWrite) { return false; } diff --git a/source/CapriKit.AssetPipeline/HotReloadManager.cs b/source/CapriKit.AssetPipeline/HotReloadManager.cs index 5e1a5e5..aff1727 100644 --- a/source/CapriKit.AssetPipeline/HotReloadManager.cs +++ b/source/CapriKit.AssetPipeline/HotReloadManager.cs @@ -5,7 +5,7 @@ using System.Collections.Concurrent; using System.Diagnostics; -namespace CapriKit.AssetPipeline.v2; +namespace CapriKit.AssetPipeline; /// /// Facilitates hot reloading and hot swapping of assets. Tracks the files used to create an asset @@ -17,7 +17,7 @@ internal sealed partial class HotReloadManager : IDisposable private static readonly TimeSpan MinWaitTime = TimeSpan.FromSeconds(0.5); private readonly ILogger Logger; - private readonly IVirtualFileSystem FileSystem; + private readonly ScopedFileSystem FileSystem; private readonly IVirtualFileSystemWatcher Watcher; private readonly FileSystemEventQueue FileChances; @@ -30,12 +30,12 @@ internal sealed partial class HotReloadManager : IDisposable private long lastFileChange; private bool isReloading; - public HotReloadManager(ILoggerFactory logger, IVirtualFileSystem fileSystem) + public HotReloadManager(ILoggerFactory logger, ScopedFileSystem fileSystem) { Logger = logger.CreateLogger(); FileSystem = fileSystem; - Watcher = fileSystem.Watch(DirectoryPath.Empty); // Watch for all changed, usually fileSystem is a ScopedVirtualFileSystem + Watcher = fileSystem.Watch(); FileChances = new FileSystemEventQueue(Watcher); Tracked = []; diff --git a/source/CapriKit.DirectX11/Resources/Shaders/ShaderIncludeResolver.cs b/source/CapriKit.DirectX11/Resources/Shaders/ShaderIncludeResolver.cs index c81643c..b376350 100644 --- a/source/CapriKit.DirectX11/Resources/Shaders/ShaderIncludeResolver.cs +++ b/source/CapriKit.DirectX11/Resources/Shaders/ShaderIncludeResolver.cs @@ -12,11 +12,13 @@ private sealed class ShaderStream(FilePath source, byte[] buffer) : MemoryStream public FilePath Source { get; } = source; } - private readonly ReadOnlyScopedFileSystem FileSystem; + private readonly IReadOnlyVirtualFileSystem FileSystem; + private readonly DirectoryPath BasePath; public ShaderIncludeResolver(IReadOnlyVirtualFileSystem fileSystem, DirectoryPath basePath) { - FileSystem = new ReadOnlyScopedFileSystem(fileSystem, basePath); + FileSystem = fileSystem; + BasePath = fileSystem.GetAbsolutePath(basePath); } public Stream Open(IncludeType type, string fileName, Stream? parentStream) @@ -28,15 +30,16 @@ public Stream Open(IncludeType type, string fileName, Stream? parentStream) fileName = includer.Source.Directory.Append(fileName); } + var path = BasePath.Append(fileName); // The stream might be UTF-8 or UTF-16 even if the contents // are only ASCII characters. Read the full file and convert // it before passing it to DirectX. - using var fileStream = FileSystem.OpenRead(fileName); + using var fileStream = FileSystem.OpenRead(path); using var reader = new StreamReader(fileStream); var text = reader.ReadToEnd(); var bytes = Encoding.ASCII.GetBytes(text); - return new ShaderStream(fileName, bytes); + return new ShaderStream(path, bytes); } public void Close(Stream stream) diff --git a/source/CapriKit.IO/DirectoryPath.cs b/source/CapriKit.IO/DirectoryPath.cs index 0477c0f..aeee232 100644 --- a/source/CapriKit.IO/DirectoryPath.cs +++ b/source/CapriKit.IO/DirectoryPath.cs @@ -38,17 +38,6 @@ public DirectoryPath? Parent } } - public DirectoryPath ToAbsolute() - { - if (IsAbsolute) - { - return this; - } - - var full = System.IO.Path.GetFullPath(Path); - return new DirectoryPath(full); - } - public DirectoryPath ToAbsolute(DirectoryPath basePath) { if (IsAbsolute) diff --git a/source/CapriKit.IO/FilePath.cs b/source/CapriKit.IO/FilePath.cs index 162ea6f..ed25bcc 100644 --- a/source/CapriKit.IO/FilePath.cs +++ b/source/CapriKit.IO/FilePath.cs @@ -31,8 +31,6 @@ public FilePath(ReadOnlySpan path) public bool IsAbsolute => System.IO.Path.IsPathFullyQualified(Path); - public FilePath ToAbsolute() => new(System.IO.Path.GetFullPath(Path)); - public FilePath ToAbsolute(DirectoryPath basePath) { if (IsAbsolute) diff --git a/source/CapriKit.IO/FileSystem.cs b/source/CapriKit.IO/FileSystem.cs index 932290f..2a34865 100644 --- a/source/CapriKit.IO/FileSystem.cs +++ b/source/CapriKit.IO/FileSystem.cs @@ -92,6 +92,10 @@ public IVirtualFileSystemWatcher Watch(DirectoryPath directory, bool includeSubD return new FileSystemEventListener(directory, includeSubDirectories); } + public FilePath GetAbsolutePath(FilePath file) => file.ToAbsolute(Environment.CurrentDirectory); + + public DirectoryPath GetAbsolutePath(DirectoryPath directory) => directory.ToAbsolute(Environment.CurrentDirectory); + private FileInfo FindOrThrow(FilePath file) { var info = GetFileInfo(file); @@ -110,13 +114,13 @@ internal static FilePath GetFilePath(string path) internal static FileInfo GetFileInfo(FilePath file) { - var absolutePath = file.IsAbsolute ? file : file.ToAbsolute(); + var absolutePath = file.IsAbsolute ? file : file.ToAbsolute(Environment.CurrentDirectory); return new FileInfo(absolutePath.ToString()); } internal static DirectoryInfo GetDirectoryInfo(DirectoryPath path) { - var absolutePath = path.IsAbsolute ? path : path.ToAbsolute(); + var absolutePath = path.IsAbsolute ? path : path.ToAbsolute(Environment.CurrentDirectory); return new DirectoryInfo(absolutePath.ToString()); } } diff --git a/source/CapriKit.IO/IVirtualFileSystem.cs b/source/CapriKit.IO/IVirtualFileSystem.cs index 9ae8b6d..8ffc761 100644 --- a/source/CapriKit.IO/IVirtualFileSystem.cs +++ b/source/CapriKit.IO/IVirtualFileSystem.cs @@ -51,4 +51,16 @@ public interface IReadOnlyVirtualFileSystem /// Watches for changes in the given subdirectory. /// IVirtualFileSystemWatcher Watch(DirectoryPath directory, bool includeSubDirectories = true); + + /// + /// Determines the absolute path of the given relative path, depending on the file system type + /// this can be resolved using the CWD or another method. + /// + DirectoryPath GetAbsolutePath(DirectoryPath directory); + + /// + /// Determines the absolute path of the given relative path, depending on the file system type + /// this can be resolved using the CWD or another method. + /// + FilePath GetAbsolutePath(FilePath filePath); } diff --git a/source/CapriKit.IO/InMemoryFileSystem.cs b/source/CapriKit.IO/InMemoryFileSystem.cs index 9a2ab68..534e0fb 100644 --- a/source/CapriKit.IO/InMemoryFileSystem.cs +++ b/source/CapriKit.IO/InMemoryFileSystem.cs @@ -95,6 +95,10 @@ public IVirtualFileSystemWatcher Watch(DirectoryPath directory, bool includeSubD return watcher; } + public FilePath GetAbsolutePath(FilePath file) => file.ToAbsolute(Environment.CurrentDirectory); + + public DirectoryPath GetAbsolutePath(DirectoryPath directory) => directory.ToAbsolute(Environment.CurrentDirectory); + private InMemoryFile FindOrThrow(FilePath file) { if (Disk.TryGetValue(file, out var value)) diff --git a/source/CapriKit.IO/ReadOnlyVirtualFileSystemSpy.cs b/source/CapriKit.IO/ReadOnlyVirtualFileSystemSpy.cs index bc3f910..ec1bca9 100644 --- a/source/CapriKit.IO/ReadOnlyVirtualFileSystemSpy.cs +++ b/source/CapriKit.IO/ReadOnlyVirtualFileSystemSpy.cs @@ -51,4 +51,8 @@ public IVirtualFileSystemWatcher Watch(DirectoryPath directory, bool includeSubD { return Actual.Watch(directory, includeSubDirectories); } + + public FilePath GetAbsolutePath(FilePath file) => Actual.GetAbsolutePath(file); + + public DirectoryPath GetAbsolutePath(DirectoryPath directory) => Actual.GetAbsolutePath(directory); } diff --git a/source/CapriKit.IO/ScopedFileSystem.cs b/source/CapriKit.IO/ScopedFileSystem.cs index f909e65..2146964 100644 --- a/source/CapriKit.IO/ScopedFileSystem.cs +++ b/source/CapriKit.IO/ScopedFileSystem.cs @@ -46,7 +46,14 @@ public class ReadOnlyScopedFileSystem : IReadOnlyVirtualFileSystem public ReadOnlyScopedFileSystem(IReadOnlyVirtualFileSystem source, DirectoryPath basePath) { Source = source; - BasePath = basePath.ToAbsolute(); + if (basePath.IsAbsolute) + { + BasePath = basePath; + } + else + { + BasePath = source.GetAbsolutePath(basePath); + } } @@ -102,7 +109,7 @@ protected DirectoryPath GetDirectoryPath(DirectoryPath path) return path; } - var fullPath = path.GetPathRelativeTo(BasePath); + var fullPath = BasePath.Append([path]); ThrowIfPathIsOutsideBasePath(fullPath); return fullPath; @@ -116,7 +123,7 @@ protected FilePath GetFilePath(FilePath path) return path; } - var fullPath = path.GetPathRelativeTo(BasePath); + var fullPath = BasePath.Append(path); ThrowIfPathIsOutsideBasePath(fullPath); return fullPath; @@ -142,8 +149,8 @@ protected void ThrowIfPathIsOutsideBasePath(DirectoryPath path) public IVirtualFileSystemWatcher Watch(DirectoryPath directory, bool includeSubDirectories = true) { - ThrowIfPathIsOutsideBasePath(directory); var fullPath = GetAbsolutePath(directory); + ThrowIfPathIsOutsideBasePath(fullPath); var watchers = Source.Watch(fullPath, includeSubDirectories); return new ScopedFileSystemEventListener(watchers, BasePath); } diff --git a/source/CapriKit.IO/VirtualFileSystemSpy.cs b/source/CapriKit.IO/VirtualFileSystemSpy.cs index e02bdc9..18d840a 100644 --- a/source/CapriKit.IO/VirtualFileSystemSpy.cs +++ b/source/CapriKit.IO/VirtualFileSystemSpy.cs @@ -71,4 +71,8 @@ public IVirtualFileSystemWatcher Watch(DirectoryPath directory, bool includeSubD { return Actual.Watch(directory, includeSubDirectories); } + + public FilePath GetAbsolutePath(FilePath file) => Actual.GetAbsolutePath(file); + + public DirectoryPath GetAbsolutePath(DirectoryPath directory) => Actual.GetAbsolutePath(directory); } diff --git a/source/CapriKit.Tests/AssetPipeline/AssetManagerTests.cs b/source/CapriKit.Tests/AssetPipeline/AssetManagerTests.cs new file mode 100644 index 0000000..00f0044 --- /dev/null +++ b/source/CapriKit.Tests/AssetPipeline/AssetManagerTests.cs @@ -0,0 +1,10 @@ +using System; +using System.Collections.Generic; +using System.Text; + +namespace CapriKit.Tests.AssetPipeline; + +internal class AssetManagerTests +{ + +} diff --git a/source/CapriKit.Tests/AssetPipeline/HotReloadManagerTests.cs b/source/CapriKit.Tests/AssetPipeline/HotReloadManagerTests.cs new file mode 100644 index 0000000..3cfd25b --- /dev/null +++ b/source/CapriKit.Tests/AssetPipeline/HotReloadManagerTests.cs @@ -0,0 +1,15 @@ +using CapriKit.AssetPipeline.v2; +using CapriKit.IO; +using Microsoft.Extensions.Logging.Abstractions; + +namespace CapriKit.Tests.AssetPipeline; + +internal class HotReloadManagerTests +{ + [Test] + public async Task Foo() + { + var fileSystem = new InMemoryFileSystem().ScopedTo("C:/Test"); + var sut = new HotReloadManager(NullLoggerFactory.Instance, fileSystem); + } +} diff --git a/source/CapriKit.Tests/CapriKit.Tests.csproj b/source/CapriKit.Tests/CapriKit.Tests.csproj index d345c16..78a40ca 100644 --- a/source/CapriKit.Tests/CapriKit.Tests.csproj +++ b/source/CapriKit.Tests/CapriKit.Tests.csproj @@ -13,6 +13,7 @@ + diff --git a/source/CapriKit.Tests/IO/ScopedFileSystemTests.cs b/source/CapriKit.Tests/IO/ScopedFileSystemTests.cs index 4ad2a7e..72ebdf6 100644 --- a/source/CapriKit.Tests/IO/ScopedFileSystemTests.cs +++ b/source/CapriKit.Tests/IO/ScopedFileSystemTests.cs @@ -19,4 +19,18 @@ await Assert.That(() => sut.Exists(forbiddenPath); }).Throws(); } + + + [Test] + public async Task GetAbsolutePath() + { + var basePath = new DirectoryPath("C:/Temp"); + var fileSystem = new InMemoryFileSystem(); + var sut = new ScopedFileSystem(fileSystem, basePath); + + var relativePath = new FilePath("file.txt"); + var expected = new FilePath("C:/Temp/file.txt"); + + await Assert.That(sut.GetAbsolutePath(relativePath)).IsEqualTo(expected); + } } From 36a68ad0ef5151dd107e1b58ba7e5e0be423d604 Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Mon, 10 Aug 2026 00:01:32 +0200 Subject: [PATCH 29/53] WIP: embrace threading --- source/CapriKit.AssetPipeline/Asset.cs | 2 +- source/CapriKit.AssetPipeline/AssetManager.cs | 1 - source/CapriKit.AssetPipeline/TODO.md | 7 + .../vNext/AssetCache.cs | 180 ++++++++++++++++++ .../vNext/AssetManager.cs | 22 +++ 5 files changed, 210 insertions(+), 2 deletions(-) create mode 100644 source/CapriKit.AssetPipeline/TODO.md create mode 100644 source/CapriKit.AssetPipeline/vNext/AssetCache.cs create mode 100644 source/CapriKit.AssetPipeline/vNext/AssetManager.cs diff --git a/source/CapriKit.AssetPipeline/Asset.cs b/source/CapriKit.AssetPipeline/Asset.cs index bf4033d..fef278d 100644 --- a/source/CapriKit.AssetPipeline/Asset.cs +++ b/source/CapriKit.AssetPipeline/Asset.cs @@ -12,6 +12,6 @@ public record AssetId(string Key, FilePath Path); internal record Asset(AssetId Id, TAsset Value, AssetBuildMetaData BuildMetaData) where TAsset : class; -internal record class AssetBuildMetaData(Guid TranscoderId, int TranscoderVersion, TSettings Settings, IReadOnlyList Dependencies); +internal record class AssetBuildMetaData(Guid TranscoderId, int TranscoderVersion, TSettings Settings, FilePath OutputFile, IReadOnlyList Dependencies); internal sealed record Dependency(FilePath File, DateTime Version); diff --git a/source/CapriKit.AssetPipeline/AssetManager.cs b/source/CapriKit.AssetPipeline/AssetManager.cs index a999a9f..71465a6 100644 --- a/source/CapriKit.AssetPipeline/AssetManager.cs +++ b/source/CapriKit.AssetPipeline/AssetManager.cs @@ -1,4 +1,3 @@ -using CapriKit.AssetPipeline.v2; using CapriKit.IO; using Microsoft.Extensions.Logging; using System.Buffers; diff --git a/source/CapriKit.AssetPipeline/TODO.md b/source/CapriKit.AssetPipeline/TODO.md new file mode 100644 index 0000000..420adc0 --- /dev/null +++ b/source/CapriKit.AssetPipeline/TODO.md @@ -0,0 +1,7 @@ +# TODO + +- What about using channels? +- What if reloading builds to a random file, so you don't need syncing ever, and the main thread triggers moving the file to the right place. +- Instead of documentation -> Throw early +- Look at bot-review +- How top opinionate more diff --git a/source/CapriKit.AssetPipeline/vNext/AssetCache.cs b/source/CapriKit.AssetPipeline/vNext/AssetCache.cs new file mode 100644 index 0000000..9e4b64f --- /dev/null +++ b/source/CapriKit.AssetPipeline/vNext/AssetCache.cs @@ -0,0 +1,180 @@ +using System.Diagnostics.CodeAnalysis; + +namespace CapriKit.AssetPipeline.vNext; + +/// +/// Cache of live assets. Methods are thread-safe and can be accessed concurrently. However, +/// cleaning-up unused asset using the must be done from the main thread. +/// +internal sealed partial class AssetCache : IDisposable +{ + private sealed class Entry(AssetId id, object asset, int refCount) + { + public AssetId Id { get; } = id; + public object Asset { get; } = asset; + public int RefCount { get; set; } = refCount; + } + + private readonly Lock Lock = new(); + private readonly Dictionary Entries = []; + private readonly Queue PendingDispose = []; + private bool isDisposed; + + /// + /// Stores the given asset and then leases it. If another caller already stored the asset + /// the is disposed and the stored instance is leased instead. + /// Thread-safe: loading the same asset twice is wasteful but harmless, after calling this + /// method users must stop referencing . + /// + public TAsset PutOrLease(AssetId id, TAsset candidate) + where TAsset : class + { + lock (Lock) + { + ObjectDisposedException.ThrowIf(isDisposed, this); + + if (Entries.TryGetValue(id, out var entry)) + { + PendingDispose.Enqueue(new Entry(id, candidate, 0)); + + var asset = Cast(entry, id); + entry.RefCount++; + return asset; + } + + Entries.Add(id, new Entry(id, candidate, 1)); + return candidate; + } + } + + /// + /// Attempts to retrieve the asset with the given id. Thread-safe. + /// + public bool TryLease(AssetId id, [NotNullWhen(true)] out TAsset? asset) + where TAsset : class + { + lock (Lock) + { + ObjectDisposedException.ThrowIf(isDisposed, this); + + if (Entries.TryGetValue(id, out var entry)) + { + asset = Cast(entry, id); + entry.RefCount++; + return true; + } + + asset = default; + return false; + } + } + + /// + /// Returns a leased asset. If every user returned their asset it become collectable. Which happens in + /// . After calling return the caller must no longer reference the asset instance. + /// Thread-safe. + /// + public void Return(AssetId id) + { + lock (Lock) + { + ObjectDisposedException.ThrowIf(isDisposed, this); + + if (Entries.TryGetValue(id, out var entry)) + { + entry.RefCount--; + // Evict items immediately, but only dispose of them in Collect + if (entry.RefCount <= 0) + { + Entries.Remove(id); + PendingDispose.Enqueue(entry); + } + } + else + { + throw new InvalidOperationException($"Returned {id} which was not found in the cache."); + } + } + } + + /// + /// Disposes all assets that no longer have users. This method must be called from the primary thread, + /// but it is safe for threads to concurrently access other methods in this class. + /// + public void Collect() + { + List? toDispose = null; + + // Collecting usually is a no-op and runs on the most important thread,so avoid waiting on acquiring the lock. + if (Lock.TryEnter()) + { + try + { + if (isDisposed) { return; } + toDispose = DrainPendingDisposeQueue(); + } + finally + { + Lock.Exit(); + } + } + + DisposeDrainedItems(toDispose); + } + + public void Dispose() + { + int leaked; + List? toDispose; + + lock (Lock) + { + if (isDisposed) { return; } + isDisposed = true; + + leaked = Entries.Count; + Entries.Clear(); + + toDispose = DrainPendingDisposeQueue(); + } + + DisposeDrainedItems(toDispose); + + if (leaked > 0) + { + throw new Exception($"Cache will leak {leaked} entries that have not been returned before the cache was disposed."); + } + } + + // Must be called from inside a lock, returns null if there's nothing to dispose + private List? DrainPendingDisposeQueue() + { + List? toDispose = null; + while (PendingDispose.TryDequeue(out var entry)) + { + (toDispose ??= []).Add(entry); + } + return toDispose; + } + + // Must called from outside a lock. Dispose runs code outside of our control, might run long + // and might even interact with the asset cache (in which case running it inside the lock + // would cause deadlocks). + private static void DisposeDrainedItems(List? toDispose) + { + if (toDispose != null) + { + foreach (var entry in toDispose) + { + (entry.Asset as IDisposable)?.Dispose(); + } + } + } + + private static TAsset Cast(Entry entry, AssetId id) + where TAsset : class + { + return entry.Asset as TAsset + ?? throw new InvalidOperationException($"{id} is cached as {entry.Asset.GetType().Name} but was requested as {typeof(TAsset).Name}."); + } +} diff --git a/source/CapriKit.AssetPipeline/vNext/AssetManager.cs b/source/CapriKit.AssetPipeline/vNext/AssetManager.cs new file mode 100644 index 0000000..ffea3eb --- /dev/null +++ b/source/CapriKit.AssetPipeline/vNext/AssetManager.cs @@ -0,0 +1,22 @@ +using System; +using System.Collections.Generic; +using System.Text; + +namespace CapriKit.AssetPipeline.vNext; + +internal sealed partial class AssetManager : IDisposable +{ + + + public bool TryLoad(AssetId id, IAssetTranscoder transcoder, TSetting settings) + where TAsset : class + { + if() + } + + + public void Dispose() + { + throw new NotImplementedException(); + } +} From c8d19aa2c854d4d04d1944589bad7a70d8285a2f Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Mon, 10 Aug 2026 21:41:13 +0200 Subject: [PATCH 30/53] WIP: promsies --- Research/AssetPipelineLoadingGroups.md | 298 ++++++++++++++++++ source/CapriKit.AssetPipeline/Asset.cs | 2 +- .../vNext/AssetBundle.cs | 71 +++++ .../vNext/AssetManager.cs | 9 +- .../{Async => Primitives}/Drain.cs | 4 +- source/CapriKit.Tests.Tool/Program.cs | 1 + 6 files changed, 377 insertions(+), 8 deletions(-) create mode 100644 Research/AssetPipelineLoadingGroups.md create mode 100644 source/CapriKit.AssetPipeline/vNext/AssetBundle.cs rename source/CapriKit.Concurrency/{Async => Primitives}/Drain.cs (89%) diff --git a/Research/AssetPipelineLoadingGroups.md b/Research/AssetPipelineLoadingGroups.md new file mode 100644 index 0000000..d5413b2 --- /dev/null +++ b/Research/AssetPipelineLoadingGroups.md @@ -0,0 +1,298 @@ +# Asset Pipeline — Loading Groups + +Status: sketch for discussion, 2026-08-10. Companion to `AssetPipelineArchitecture.md`. + +## Context + +While working on `source/CapriKit.AssetPipeline/vNext/` — the threading-aware rewrite. `AssetCache` +was already done (thread-safe lease/return with deferred disposal); the open question was how +assets get *delivered* now that `Load` no longer returns a `Task` to gameplay code. + +The concrete worry: in the old engine a `Car` took its texture, model and shader through +constructor injection. With an optimistic (`TryLoad`) or channel-based model, every system would +seem to need a "do I have everything yet?" check in `Update`, every frame. That is the thing this +document is trying to avoid. + +## The framing + +The question isn't *task vs channel vs optimistic*. It's **who is allowed to exist before their +assets exist**. Engines don't answer that uniformly — they split the world: + +- **Engine assets** — lighting/shadow shaders, BRDF LUT, blue noise, error textures. Part of the + engine build, not content. Must exist or the engine is broken. Loaded in a boot phase *before* + any system is constructed. Constructor injection, no nulls, no polling. +- **Content assets** — levels, entities, characters. Their consumers have to tolerate absence + anyway (streaming, LOD, missing file, hot reload), so absence is designed in rather than bolted on. + +Source calls this `Precache*` at level load — requesting a non-precached asset at runtime is a hard +error in dev builds. Unreal splits `FStreamableManager` requests from always-loaded cooked packages. + +The rule that falls out: **never expose "the asset might not be here" to gameplay code.** That is +precisely what forces a per-frame check into every system. `TryLoad` is fine as an internal +cache-hit fast path; it is a trap as the public API. + +## Options considered + +| Option | Shape | Good for | Bad for | +|---|---|---|---| +| **A** Resolve-then-construct | Boot phase batch-loads declared requirements in parallel, systems are constructed from a resolved lookup that cannot fail | Lighting, Shadowing, engine systems | Anything streamed — loading the whole game up front | +| **B** Placeholder + `HotSwap` | `Load` returns a usable instance immediately whose contents are a placeholder; swapped in place when the payload lands | Assets whose absence only affects pixels | Simulation data (placeholder collision mesh = fall through floor), and **shaders** — see below | +| **C** Group barrier + completion channel | Interlocked counter per group; the worker that drives it to zero pushes the group onto a channel the main thread drains | Levels, entities, anything with N dependencies | Nothing much — it is the general mechanism | +| **D** Async confined to load scopes | Workers `await` N assets, construct the finished object, hand it to the sim through one channel | Level loading, streaming coordinators | — | + +Tasks were never the mistake; handing a `Task` to *gameplay* was, because async then infects +everything upward. In D, `Car`'s constructor still takes its assets directly — it just isn't the +main thread calling it. + +**Recommended split for CapriKit:** A for engine systems, B for renderer content, C+D for levels +and entities. + +## Why shaders can't use placeholders (Option B) + +A placeholder is a valid substitute only when **every instance of the asset type shares one +interface**. A texture qualifies: an SRV is an SRV, the consumer doesn't care what pixels are +behind it. Content varies, interface doesn't. + +For a shader the interface *is* the identity: + +- `CreateInputLayout` validates the `InputElementDescription[]` against the VS input signature in the blob +- cbuffer register assignments (`b0`, `b1`, …) and their field layouts +- SRV/sampler slots (`t0`, `s0`) + +A placeholder VS substitutes for a real VS only if it has the same input signature *and* binding +layout — at which point it is a hand-written stub per shader, not a placeholder. + +### The same gap is a latent hot-reload bug + +`IVertexShader.HotSwap` (`source/CapriKit.DirectX11/Resources/Shaders/VertexShader.cs:12`) swaps +`Blob` and `ID3D11VertexShader`. But the `IInputLayout` was created from the *old* blob and is owned +by the consumer — e.g. `ImGuiEffect.cs:24` holds it in a separate field. Nothing tells that layout +its shader changed. + +D3D11 won't crash: `CreateInputLayout` validates at creation and the layout is independent +afterwards. But reload a shader whose signature grew a `TEXCOORD1` and the old layout no longer +feeds it — debug-layer complaint, zero/garbage in that register. Subtly wrong, which is worse. + +`HotSwap` on a bare vertex shader is incoherent *because* a bare vertex shader isn't independently +usable — the same premise as the placeholder problem. + +### Fix: the effect is the atomic asset + +The transcoder's `TAsset` should be the smallest **self-consistent** unit — the thing owning a +complete, valid interface: + +```csharp +internal sealed class Effect // IAssetTranscoder produces this +{ + public ID3D11VertexShader Vertex { get; private set; } + public ID3D11PixelShader Pixel { get; private set; } + public IInputLayout InputLayout { get; private set; } // built from *this* blob, at decode + + // snip +} + +public override void HotSwap(Effect instance, Effect newParts) +{ + var old = instance.Exchange(newParts); // shader + layout swap together, main thread + // snip: dispose old +} +``` + +A mismatched (blob, layout) pair becomes unrepresentable — the layout is created inside `Decode` +from the blob being decoded. `CapriKit.Generators.HLSL` already emits the input element +descriptions as compile-time constants, so this construction is deterministic. + +Shaders then belong in **category A**: they're a few KB, the set is closed at compile time (the +generator enumerates every `#pragma VertexShader` / `#pragma PixelShader` entry point, so the boot +manifest can be generated), and a renderer missing its lighting shader isn't degraded, it's +non-functional. + +For content-driven materials the answer is **skip the draw**, not substitute: the renderable isn't +registered with the render system until its effect exists. Nothing to branch on in the hot loop — +the object simply isn't in the visible set. That's UE5's PSO-precache behaviour, and it composes +with the group barrier below. + +A real shader fallback is achievable but only under a convention: fix the binding layout +engine-wide (per-frame `b0`, per-view `b1`, per-object `b2`) and standardize vertex formats. Then a +magenta error shader with a matching input signature is a genuine drop-in — that is exactly why +Unity's error shader works, it demands only the minimal interface. Worth building for the +**broken/corrupt** case so a bad edit doesn't kill the frame; not worth it for the not-yet-loaded +case, which boot-loading eliminates. + +## Option C sketch — typed loading groups + +Assumption: requirements are hardcoded in C# for the foreseeable future. + +### The type-safety trick + +Make "not loaded yet" **unrepresentable** in the consumer's types. Three types do it: + +- `Ticket` — a claim check with *no value accessor at all* +- `ResolvedAssets` — the only way to redeem a ticket, obtainable only after the group completes +- `AssetGroup` — produces one strongly-typed result whose fields are plain non-nullable assets + +No `IsLoaded`, no `.Value` that can be read early, and no arity explosion +(`AssetGroup`) because tickets carry their type individually and a closure +reassembles them. + +### Declaration side + +```csharp +public abstract class Ticket +{ + internal AssetId Id { get; init; } + internal object? Asset; // written by loader, read on main thread — the one erasure point +} + +/// A claim check for one asset. Redeem with . +public sealed class Ticket : Ticket where TAsset : class; + +public readonly struct ResolvedAssets +{ + // Safe by construction: the ticket's loader produced exactly TAsset + public TAsset Get(Ticket ticket) where TAsset : class + => (TAsset)ticket.Asset!; +} + +public sealed class AssetGroupBuilder +{ + public Ticket Add( + AssetId id, IAssetTranscoder transcoder, TSettings settings) + where TAsset : class + { + var ticket = new Ticket { Id = id }; + // Both type params erased into the closure; TAsset survives on the ticket + Requests.Add(new Request(ticket, (m, ct) => m.LoadAsync(id, transcoder, settings, ct))); + return ticket; + } + + public AssetGroup Build(Func factory) { /* snip */ } +} +``` + +### A hardcoded requirement list + +```csharp +internal sealed class ShadowAssets +{ + public static AssetGroup Define( + AssetGroupBuilder b, EffectTranscoder effects, TextureTranscoder textures) + { + var shadow = b.Add("shaders/shadow.hlsl", effects, EffectSettings.Default); + var noise = b.Add("textures/blue-noise.png", textures, TextureSettings.Default); + + return b.Build(r => new ShadowAssets(r.Get(shadow), r.Get(noise))); + } + + private ShadowAssets(Effect shadow, ITexture2D noise) + { + Shadow = shadow; + Noise = noise; + } + + public Effect Shadow { get; } // non-nullable, always valid + public ITexture2D Noise { get; } +} + +// The system never sees the pipeline at all +internal sealed class ShadowSystem(ShadowAssets assets, Device device) { /* snip */ } +``` + +Adding a requirement is two lines and the compiler forces the constructor to match. Deleting one +breaks the build. That is the payoff for hardcoding requirements. + +### The barrier and the drain + +```csharp +public abstract class AssetGroup : IDisposable +{ + private int outstanding; + + // Called on whichever worker finished the request + internal void OnRequestCompleted(ChannelWriter ready) + { + if (Interlocked.Decrement(ref outstanding) == 0) { ready.TryWrite(this); } + } + + internal abstract void Materialize(); // main thread only +} + +public sealed class AssetGroup : AssetGroup +{ + private readonly Func Factory; + private readonly TaskCompletionSource Source = + new(TaskCreationOptions.RunContinuationsAsynchronously); // don't hijack the main thread + + internal override void Materialize() => Source.SetResult(Factory(new ResolvedAssets())); + + public Task Completion => Source.Task; + public bool TryTake([NotNullWhen(true)] out TResult? value) { /* snip */ } +} +``` + +```csharp +// AssetManager.Update() — main thread, once per frame +while (ReadyGroups.Reader.TryRead(out var group)) +{ + group.Materialize(); // usually zero iterations +} +Cache.Collect(); +``` + +This answers the original worry: the main thread drains **one** channel of finished groups. No +system polls its own assets, and the per-frame cost is O(groups that finished this frame), normally +zero. + +One mechanism serves both consumption styles: + +```csharp +// Boot / loading screen (Option A) +var group = ShadowAssets.Define(builder, effects, textures); +services.AddSingleton(new ShadowSystem(await group.Completion, device)); + +// Streaming (Option C) — entity doesn't exist until the group lands +if (pendingLevel.TryTake(out var level)) { World.Install(level); } +``` + +### Lifetime and failure + +Make the **group** the refcount unit, not the individual asset — it lines up with +`AssetCache.Return` and makes level unload a single `Dispose`: + +```csharp +public void Dispose() +{ + foreach (var ticket in Tickets) + { + if (ticket.Asset is not null) { Cache.Return(ticket.Id); } + } +} +``` + +The failure path matters because `AssetCache.Dispose` throws on leaks. If request 4 of 5 throws, +the three that already landed hold leases. Catch per-request, store the exception, let the counter +reach zero anyway, and have `Materialize` fault the group *and* return the partial leases. +Otherwise one bad asset file becomes a leak exception at shutdown that says nothing about the cause. + +## Constraints to document for transcoder authors + +**Where `Decode` may run.** `Device.cs:21-31` creates the device without +`DeviceCreationFlags.Singlethreaded`, so `ID3D11Device` resource creation is free-threaded — +`CreateVertexShader` / `CreateTexture2D` on a worker is fine. `ID3D11DeviceContext` is **not**. A +transcoder needing `Map` or `UpdateSubresource` must either create immutable resources with initial +data (no context involved) or defer that step to `Materialize` on the main thread. Violating this +produces corruption rather than an exception, so it needs to be an explicit written rule. + +**Ticket provenance.** Nothing at compile time stops redeeming a ticket from group A inside group +B's factory. The closure capture makes it awkward in practice; a `Debug.Assert(ticket.Owner == this)` +in `Get` covers the rest cheaply. + +## Open questions + +- Does `AssetGroup` need `Task` at all, or is `TryTake` + a main-thread callback enough? + `Completion` is convenient for boot code that legitimately awaits on a loading screen. +- Where does hot reload re-enter? A reloaded asset inside a live group needs `HotSwap`, not + re-materialization — the `TResult` already handed out must keep its identity. +- Should `AssetManager.TryLoad` stay public? Current stub returns `bool` with nowhere to put the + asset; it wants `out TAsset` with `[NotNullWhen(true)]`, mirroring `AssetCache.TryLease` — and + arguably it should be internal, as the fast path the three policies sit on. diff --git a/source/CapriKit.AssetPipeline/Asset.cs b/source/CapriKit.AssetPipeline/Asset.cs index fef278d..bf4033d 100644 --- a/source/CapriKit.AssetPipeline/Asset.cs +++ b/source/CapriKit.AssetPipeline/Asset.cs @@ -12,6 +12,6 @@ public record AssetId(string Key, FilePath Path); internal record Asset(AssetId Id, TAsset Value, AssetBuildMetaData BuildMetaData) where TAsset : class; -internal record class AssetBuildMetaData(Guid TranscoderId, int TranscoderVersion, TSettings Settings, FilePath OutputFile, IReadOnlyList Dependencies); +internal record class AssetBuildMetaData(Guid TranscoderId, int TranscoderVersion, TSettings Settings, IReadOnlyList Dependencies); internal sealed record Dependency(FilePath File, DateTime Version); diff --git a/source/CapriKit.AssetPipeline/vNext/AssetBundle.cs b/source/CapriKit.AssetPipeline/vNext/AssetBundle.cs new file mode 100644 index 0000000..4fa71f0 --- /dev/null +++ b/source/CapriKit.AssetPipeline/vNext/AssetBundle.cs @@ -0,0 +1,71 @@ +using CapriKit.Concurrency.Primitives; + +namespace CapriKit.AssetPipeline.vNext; + +public interface IPromise +{ + public TKey Key { get; } + internal object? Value { get; set; } + + internal int Owner { get; } +} + +public sealed class Promise(TKey key, int owner) : IPromise +{ + private TValue? _value; + + public TKey Key { get; } = key; + public int Owner { get; } = owner; + + object? IPromise.Value + { + get => _value; + set => _value = (TValue?)value; + } +} + +public sealed class PromiseResolver(int id) +{ + private readonly int Id = id; + + public T Get(IPromise promise) + { + if (promise.Owner != Id) + { + throw new InvalidOperationException($"Attempted to resolve a promise that was not owned by this resolver"); + } + + if (promise.Value is T value) + { + return value; + } + + throw new Exception($"Internal error: resolved value was not of type {typeof(T).Name} but {promise.Value?.GetType().Name ?? "null"}"); + } +} + + +public abstract class AssetBundle +{ + private int outstanding; + + internal void OnRequestCompleted(LightweightChannel ready) + { + if (Interlocked.Decrement(ref outstanding) == 0) + { + ready.Write(this); + } + } + + internal abstract void Materialize(); +} + +public sealed internal class AssetBundle : AssetBundle +{ + private readonly TaskCompletionSource Source = new(TaskCreationOptions.RunContinuationsAsynchronously); + + internal override void Materialize() + { + Source.SetResult() + } +} diff --git a/source/CapriKit.AssetPipeline/vNext/AssetManager.cs b/source/CapriKit.AssetPipeline/vNext/AssetManager.cs index ffea3eb..1aa3c2a 100644 --- a/source/CapriKit.AssetPipeline/vNext/AssetManager.cs +++ b/source/CapriKit.AssetPipeline/vNext/AssetManager.cs @@ -1,7 +1,3 @@ -using System; -using System.Collections.Generic; -using System.Text; - namespace CapriKit.AssetPipeline.vNext; internal sealed partial class AssetManager : IDisposable @@ -11,7 +7,10 @@ internal sealed partial class AssetManager : IDisposable public bool TryLoad(AssetId id, IAssetTranscoder transcoder, TSetting settings) where TAsset : class { - if() + // TODO: Check if the file is in the AssetCache + // TODO: Otherwise kick off the loading process + // TODO: build to random output file and only move it to the right one once the main thread drains it. + throw new NotImplementedException(); } diff --git a/source/CapriKit.Concurrency/Async/Drain.cs b/source/CapriKit.Concurrency/Primitives/Drain.cs similarity index 89% rename from source/CapriKit.Concurrency/Async/Drain.cs rename to source/CapriKit.Concurrency/Primitives/Drain.cs index e7b2717..300e2b2 100644 --- a/source/CapriKit.Concurrency/Async/Drain.cs +++ b/source/CapriKit.Concurrency/Primitives/Drain.cs @@ -1,7 +1,7 @@ -using CapriKit.Concurrency.Primitives; +using CapriKit.Concurrency.Async; using System.Diagnostics.CodeAnalysis; -namespace CapriKit.Concurrency.Async; +namespace CapriKit.Concurrency.Primitives; /// /// Allows you to drain work as it completes diff --git a/source/CapriKit.Tests.Tool/Program.cs b/source/CapriKit.Tests.Tool/Program.cs index a96ec03..1673707 100644 --- a/source/CapriKit.Tests.Tool/Program.cs +++ b/source/CapriKit.Tests.Tool/Program.cs @@ -1,4 +1,5 @@ using CapriKit.Concurrency.Async; +using CapriKit.Concurrency.Primitives; using CapriKit.DirectX11; using CapriKit.DirectX11.Debug; using CapriKit.IO; From a734cbfad49ad1bc6a8bbfd3a886f726b5c8bdbb Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Mon, 10 Aug 2026 23:10:51 +0200 Subject: [PATCH 31/53] WIP: promises and bundles --- .../vNext/AssetBundle.cs | 99 +++++++++++-------- .../vNext/AssetManager.cs | 20 +++- .../CapriKit.Concurrency/Promises/Promise.cs | 34 +++++++ .../AssetPipeline/HotReloadManagerTests.cs | 2 +- 4 files changed, 113 insertions(+), 42 deletions(-) create mode 100644 source/CapriKit.Concurrency/Promises/Promise.cs diff --git a/source/CapriKit.AssetPipeline/vNext/AssetBundle.cs b/source/CapriKit.AssetPipeline/vNext/AssetBundle.cs index 4fa71f0..c3e03d7 100644 --- a/source/CapriKit.AssetPipeline/vNext/AssetBundle.cs +++ b/source/CapriKit.AssetPipeline/vNext/AssetBundle.cs @@ -1,71 +1,90 @@ using CapriKit.Concurrency.Primitives; +using CapriKit.Concurrency.Promises; +using CapriKit.IO; +using CapriKit.IO.Streams; +using System.Buffers; namespace CapriKit.AssetPipeline.vNext; -public interface IPromise +// TODO: this and the promises in CapriKit.Concurrency implement the ideas from +// research\AssetPipelineLoadingGroups.md +// but there are still a few open questions +// - I need a blocking one so that all systems can start (Bootstrap), is that await? In CapriKit.Test.Tool I seem to be able to avoid that +// - I need a non-blocking one for all the other assets for stuff that should happen during loading screens what mechanism to use? +// - LightWeightChannel to the rescue and then swapping the entire 'loading scene' with the new scene? +// - When does the actual loading start and how do we register for it? See AssetManager.Bundle +// - How/when do we set the owner of the promise for the extra check? + +public abstract class AssetBundle { - public TKey Key { get; } - internal object? Value { get; set; } + private int outstanding; + + internal void OnRequestCompleted(LightweightChannel ready) + { + if (Interlocked.Decrement(ref outstanding) == 0) + { + ready.Write(this); + } + } - internal int Owner { get; } + internal abstract void Materialize(); } -public sealed class Promise(TKey key, int owner) : IPromise +public sealed class AssetBundle : AssetBundle { - private TValue? _value; + private readonly int Id; + private readonly Func Resolver; + private readonly TaskCompletionSource Source = new(TaskCreationOptions.RunContinuationsAsynchronously); - public TKey Key { get; } = key; - public int Owner { get; } = owner; + internal AssetBundle(int id, Func resolver) + { + Id = id; + Resolver = resolver; + } - object? IPromise.Value + internal override void Materialize() { - get => _value; - set => _value = (TValue?)value; + Source.SetResult(Resolver(new PromiseResolver(Id))); } + + public Task Completion => Source.Task; } -public sealed class PromiseResolver(int id) -{ - private readonly int Id = id; - public T Get(IPromise promise) +public sealed record ExampleAssets(object A, string B) +{ + public static AssetBundle Define(AssetManager assetManager) { - if (promise.Owner != Id) - { - throw new InvalidOperationException($"Attempted to resolve a promise that was not owned by this resolver"); - } + var transcoder = new ExampleTranscoder(); - if (promise.Value is T value) - { - return value; - } + var a = assetManager.Load(new AssetId("key", "path"), transcoder, default); + var b = assetManager.Load(new AssetId("key", "path"), transcoder, default); + + return assetManager.Bundle(r => new ExampleAssets(r.Get(a), r.Get(b))); + } - throw new Exception($"Internal error: resolved value was not of type {typeof(T).Name} but {promise.Value?.GetType().Name ?? "null"}"); + public static async Task Foo(AssetBundle bundle) + { + ExampleAssets assets = await bundle.Completion; } } -public abstract class AssetBundle +public sealed class ExampleTranscoder() : NoSettingsTranscoder(Guid.NewGuid(), 1) { - private int outstanding; - - internal void OnRequestCompleted(LightweightChannel ready) + public override Task Encode(AssetId id, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer) { - if (Interlocked.Decrement(ref outstanding) == 0) - { - ready.Write(this); - } + writer.Write("Hello World"); + return Task.CompletedTask; } - internal abstract void Materialize(); -} - -public sealed internal class AssetBundle : AssetBundle -{ - private readonly TaskCompletionSource Source = new(TaskCreationOptions.RunContinuationsAsynchronously); + public override string Decode(AssetId id, ref SequenceReader reader) + { + return reader.ReadString(); + } - internal override void Materialize() + public override void HotSwap(string instance, string newParts) { - Source.SetResult() + throw new NotImplementedException(); } } diff --git a/source/CapriKit.AssetPipeline/vNext/AssetManager.cs b/source/CapriKit.AssetPipeline/vNext/AssetManager.cs index 1aa3c2a..4a0d54c 100644 --- a/source/CapriKit.AssetPipeline/vNext/AssetManager.cs +++ b/source/CapriKit.AssetPipeline/vNext/AssetManager.cs @@ -1,6 +1,8 @@ +using CapriKit.Concurrency.Promises; + namespace CapriKit.AssetPipeline.vNext; -internal sealed partial class AssetManager : IDisposable +public sealed partial class AssetManager : IDisposable { @@ -14,8 +16,24 @@ public bool TryLoad(AssetId id, IAssetTranscoder Load(AssetId id, IAssetTranscoder transcoder, TSetting settings) + where TAsset : class + { + throw new NotImplementedException(); + } + + public void Dispose() { throw new NotImplementedException(); } + + internal AssetBundle Bundle(Func constructor) + { + throw new NotImplementedException(); + + // TODO: do something so that bundle gets notified when something is done loading + // or maybe the actual loading should only start here? + return new AssetBundle(1, constructor); + } } diff --git a/source/CapriKit.Concurrency/Promises/Promise.cs b/source/CapriKit.Concurrency/Promises/Promise.cs new file mode 100644 index 0000000..3e0574d --- /dev/null +++ b/source/CapriKit.Concurrency/Promises/Promise.cs @@ -0,0 +1,34 @@ +namespace CapriKit.Concurrency.Promises; + +public abstract class Promise +{ + internal Promise(int owner) + { + Owner = owner; + } + + internal int Owner { get; } + internal object? Value { get; set; } +} + +public sealed class Promise(int owner) : Promise(owner); + +public sealed class PromiseResolver(int id) +{ + private readonly int Id = id; + + public TValue Get(Promise promise) + { + if (promise.Owner != Id) + { + throw new InvalidOperationException($"Attempted to resolve a promise that was not owned by this resolver"); + } + + if (promise.Value is TValue value) + { + return value; + } + + throw new Exception($"Internal error: resolved value was not of type {typeof(TValue).Name} but {promise.Value?.GetType().Name ?? "null"}"); + } +} diff --git a/source/CapriKit.Tests/AssetPipeline/HotReloadManagerTests.cs b/source/CapriKit.Tests/AssetPipeline/HotReloadManagerTests.cs index 3cfd25b..4200c26 100644 --- a/source/CapriKit.Tests/AssetPipeline/HotReloadManagerTests.cs +++ b/source/CapriKit.Tests/AssetPipeline/HotReloadManagerTests.cs @@ -1,4 +1,4 @@ -using CapriKit.AssetPipeline.v2; +using CapriKit.AssetPipeline; using CapriKit.IO; using Microsoft.Extensions.Logging.Abstractions; From 176b9baacbadcc60b96da944e26369424706cb09 Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Tue, 11 Aug 2026 20:21:12 +0200 Subject: [PATCH 32/53] WIP: ARG --- Research/AssetPipelineLoadingGroupsV2.md | 273 ++++++++++++++++++ .../vNext/AssetBundle.cs | 72 ++--- .../vNext/AssetManager.cs | 34 ++- .../CapriKit.AssetPipeline/vNext/Promise.cs | 27 ++ .../CapriKit.Concurrency/Promises/Promise.cs | 34 --- 5 files changed, 351 insertions(+), 89 deletions(-) create mode 100644 Research/AssetPipelineLoadingGroupsV2.md create mode 100644 source/CapriKit.AssetPipeline/vNext/Promise.cs delete mode 100644 source/CapriKit.Concurrency/Promises/Promise.cs diff --git a/Research/AssetPipelineLoadingGroupsV2.md b/Research/AssetPipelineLoadingGroupsV2.md new file mode 100644 index 0000000..7aeaf2b --- /dev/null +++ b/Research/AssetPipelineLoadingGroupsV2.md @@ -0,0 +1,273 @@ +# Asset Pipeline — Loading Groups V2 (bundles and promises) + +Status: design correction, 2026-08-11. Supersedes the Option C sketch in +`AssetPipelineLoadingGroups.md`; that document's framing (engine vs content assets, why shaders +can't use placeholders, the transcoder threading rules) still stands. + +## Context + +Continuing `source/CapriKit.AssetPipeline/vNext/`. The V1 sketch used `Task`/`await` for group +completion. That was dropped in favour of two explicit consumption styles: + +- **block** — assets the caller cannot function without (bootstrap, engine systems) +- **poll** — assets the caller can proceed without, so the main thread keeps running a loading screen + +`AssetGroup` became `AssetBundle`, `Ticket` became `Promise`, and `AssetGroupBuilder` was +dropped in favour of `AssetManager.Load` returning promises directly. + +That last change is what caused the deadlock in the design: + +```csharp +// AssetBundle.cs — the shape that doesn't work +var a = assetManager.Load(id, default); +var b = assetManager.Load(id, default); +return assetManager.Bundle(r => new ExampleAssets(r.Get(a), r.Get(b))); +``` + +The bundle needs the promises to exist first (they're captured in the resolver closure), but +`AssetBundle.OnRequestCompleted` means each promise needs a back-pointer to the bundle. And if a +load finishes before `Bundle(...)` is called, `Update()` has nothing to signal — the countdown +never reaches zero. + +## Diagnosis + +Three problems were tangled into one. Only the first is about ordering, and it is self-inflicted. + +### 1. The cycle exists only because the bundle wants to be *told* + +`OnRequestCompleted` is a push notification. Push requires the observer to exist before the event. +Delete the notification and the ordering constraint goes with it — the bundle reads the state of its +promises instead of counting events: + +```csharp +public bool Check([MaybeNullWhen(false)] out T value) +{ + if (this.result is null) + { + foreach (var promise in this.Promises) + { + if (!promise.IsResolved) { value = default; return false; } + } + this.result = Factory(new PromiseResolver(this)); // exactly once + } + value = this.result; + return true; +} +``` + +What this removes: + +- **The chicken-and-egg.** Promises never reference the bundle while loading. Build order is free. +- **"Already loaded before the bundle existed."** A promise resolved at birth from the cache is + simply resolved; the scan sees `true`. There is no missed edge to compensate for, because the + bundle reads a *state*, not a stream of events. +- **`CountdownEvent`, `OnRequestCompleted`, and the `Bundle` half of the `OutstandingPromises` + tuple** (`AssetManager.cs:8`). + +Cost is O(n) per `Check` with n = 2–20 promises per bundle, against O(1) for the counter. Worth it. +If bundle counts ever grow into the thousands, reintroduce the counter — but it must then be +initialised at *build* time from the promises that are not yet resolved, which is the same +reconciliation done eagerly. + +### 2. The in-flight table is 1:1, but the relationship is 1:N + +`Dictionary` holds one promise per id. Two bundles can want the +same asset, and one bundle can want it twice — `ExampleAssets.Define` requests +`new AssetId("key", "path")` twice today, so the second `Load` would overwrite the first entry and +promise `a` would hang forever. This was the `// TODO: what if 2 promises wait for the same thing`. + +Make the in-flight table the dedupe point: + +```csharp +private readonly Dictionary> InFlight = []; + +internal Promise Request(AssetId id, TSettings settings) + where TAsset : class +{ + var promise = new Promise { Id = id }; + + if (Cache.TryLease(id, out var cached)) + { + promise.Value = cached; // resolved before it was ever outstanding + return promise; + } + + if (this.InFlight.TryGetValue(id, out var waiting)) { waiting.Add(promise); } + else { this.InFlight[id] = [promise]; Dispatch(id, settings); } + + return promise; +} + +public void Update() +{ + while (Ready.TryRead(out var kv)) + { + if (!this.InFlight.Remove(kv.Id, out var waiting)) { continue; } + + var asset = Cache.PutOrLease(kv.Id, kv.Asset); + foreach (var promise in waiting) { promise.Value = asset; } + // snip: take waiting.Count - 1 extra leases + } + Cache.Collect(); +} +``` + +Two requests for one id now cost one disk read and fill both promises. + +**Refcount detail:** `AssetCache.PutOrLease` takes exactly one lease. N promises that will each +`Return` once on bundle disposal need N−1 additional `TryLease` calls here, or `AssetCache.Dispose` +throws on the asymmetry. + +### 3. The builder from V1 was dropped and is still needed + +`AssetGroupBuilder` in V1 gave the group an identity before its tickets. Moving `Load` onto the +manager is what made the ordering feel impossible. Pull (fix 1) already resolves the ordering, but +the builder is still wanted for three other reasons: it defines which promises a bundle **owns** +(for `Dispose` → `Cache.Return`), it supplies the resolver-ownership check, and it makes "when does +loading start" explicit. + +```csharp +public sealed class AssetBundleBuilder(AssetManager manager) +{ + private readonly List Promises = []; + + public Promise Load(AssetId id, TSettings settings) + where TAsset : class + { + var promise = manager.Request(id, settings); + this.Promises.Add(promise); + return promise; + } + + public AssetBundle Build(Func factory) where T : notnull + { + var bundle = new AssetBundle(manager, [.. this.Promises], factory); + foreach (var promise in this.Promises) { promise.Owner = bundle; } + return bundle; + } +} +``` + +```csharp +public static AssetBundle Define(AssetManager assetManager) +{ + var builder = assetManager.CreateBundle(); + var a = builder.Load(new AssetId("a", "example.txt"), default); + var b = builder.Load(new AssetId("b", "example.txt"), default); + + return builder.Build(r => new ExampleAssets(r.Get(a), r.Get(b))); +} +``` + +Loading starts at `Load`, not at `Build` — there is no reason to defer, because an early completion +is now harmless. `Owner` is assigned in `Build`, which closes V1's open question on promise +provenance: `PromiseResolver` takes the bundle (`new PromiseResolver(this)`) and `Get` asserts +`promise.Owner == owner` rather than the commented-out check in `Promise.cs:15`. + +## The deadlock that hasn't been hit yet + +Independent of the above, and the most dangerous item here because it only appears once `Update` is +wired into the game loop. + +`AssetBundle.Wait` blocks on `Outstanding.Wait()`. `OnRequestCompleted` fires from +`AssetManager.Update()`. In the bootstrap case both are the main thread: it blocks waiting for a +signal only it could deliver. + +This forces a decision about **who applies completions**. Keep it on the main thread — that gives +lock-free `Promise.Value` and `InFlight`, and it is where a transcoder's `ID3D11DeviceContext` +finalisation step has to run anyway (see the transcoder constraints in V1). Blocking then means +*pumping*, not sleeping: + +```csharp +public T Wait(CancellationToken cancellationToken = default) +{ + var spin = new SpinWait(); + while (!Check(out var value)) + { + cancellationToken.ThrowIfCancellationRequested(); + Manager.Update(); // the waiting thread does the work instead of sleeping on it + spin.SpinOnce(); + } + return value; +} +``` + +This is the helper pattern from job systems (Unity's `JobHandle.Complete`), and it collapses both +consumption styles onto one mechanism: `Wait` is `Check` in a pumping loop, `Check` is one test +against a drain the game loop already performed. No `await`, no `CountdownEvent`, no kernel wait. + +**Constraint to document:** `Wait` and `Check` are main-thread only. If a worker should later build +a level off-thread (V1 option D), give that path a real `ManualResetEventSlim` signalled from +`Update`. + +## Two bugs in the current code, independent of the redesign + +**`Check` re-materialises the bundle.** `AssetBundle.cs:47` calls `Wait()`, which calls +`Resolver(...)`, so every successful `Check` constructs a *new* `ExampleAssets`. Polling it for 60 +frames on a loading screen yields 60 distinct instances, and hot reload has no stable object to swap +into. Materialise once and cache (`this.result` above) — this is what V1's `Materialize()` was for. + +**No faulted state.** `Promise` has only `Value`. Give it an `ExceptionDispatchInfo?` alongside, and +treat faulted as resolved. Otherwise one missing file makes `Wait` spin forever instead of throwing, +and the promises that did land leak their leases into `AssetCache.Dispose`. V1 already called this +out ("let the counter reach zero anyway"); with pull it becomes "let the promise resolve as +faulted". + +## Nullability footnote on `Check` + +The original signature `bool Check([NotNullWhen(true)] out T? value)` fails to compile under this +repo's `nullable` (`Directory.Build.props:12`) with **CS8762**, +*"Parameter 'value' must have a non-null value when exiting with 'true'"* — verified against the +SDK. + +For an unconstrained type parameter, `T` and `T?` have the same null-state: `T` could be +instantiated as `string?`, so the compiler cannot prove `Wait()` returns non-null. `[NotNullWhen]` +is a promise with nothing to back it. Either use the canonical `TryGetValue` shape +`[MaybeNullWhen(false)] out T value` (no constraint needed), or keep `where T : notnull` on +`AssetBundle`, which the current code already has and which makes the original signature legal. +`notnull` is worth keeping regardless — a resolved bundle that is null is meaningless. + +## Resulting data model + +```csharp +public abstract class Promise +{ + internal AssetId Id { get; init; } + internal object? Value; // written on the main thread only + internal ExceptionDispatchInfo? Error; // faulted counts as resolved + internal AssetBundle? Owner; // assigned in Build, for the provenance check + internal bool IsResolved => Value is not null || Error is not null; +} + +public abstract class AssetBundle; // no counter, no OnRequestCompleted + +public sealed class AssetBundle : AssetBundle where T : notnull +{ + private readonly AssetManager Manager; + private readonly Promise[] Promises; // also the Dispose → Cache.Return set + private readonly Func Factory; + private T? result; // materialised once + // snip: Check, Wait, Dispose +} + +public sealed partial class AssetManager +{ + private readonly Dictionary> InFlight = []; + private readonly LightweightChannel<(AssetId Id, object Asset)> Ready = new(); + private readonly AssetCache Cache = new(); + // snip: CreateBundle, Request, Update, Dispose +} +``` + +## Open questions + +- Cancellation: dropping a bundle mid-load leaves entries in `InFlight` whose promises nobody reads. + Harmless (the drain skips them) but it means the load still completes and takes a lease. Does + `AssetBundle.Dispose` need to prune `InFlight`, or is letting it land and immediately `Return` it + simpler? +- Hot reload re-entry is unchanged from V1 and still unanswered: a reloaded asset inside a live + bundle needs `HotSwap`, not re-materialisation, because `this.result` has already been handed out. +- Does `TryLoad` (`AssetManager.cs:12`) survive at all? With `Request` checking the cache inline it + looks redundant — V1 already suspected it should be internal. +- `SpinWait` in the `Wait` pump busy-burns a core during bootstrap. Probably fine for a few hundred + ms at startup; revisit if boot loading grows. diff --git a/source/CapriKit.AssetPipeline/vNext/AssetBundle.cs b/source/CapriKit.AssetPipeline/vNext/AssetBundle.cs index c3e03d7..865f2c9 100644 --- a/source/CapriKit.AssetPipeline/vNext/AssetBundle.cs +++ b/source/CapriKit.AssetPipeline/vNext/AssetBundle.cs @@ -1,8 +1,7 @@ -using CapriKit.Concurrency.Primitives; -using CapriKit.Concurrency.Promises; using CapriKit.IO; using CapriKit.IO.Streams; using System.Buffers; +using System.Diagnostics.CodeAnalysis; namespace CapriKit.AssetPipeline.vNext; @@ -15,76 +14,51 @@ namespace CapriKit.AssetPipeline.vNext; // - When does the actual loading start and how do we register for it? See AssetManager.Bundle // - How/when do we set the owner of the promise for the extra check? -public abstract class AssetBundle +public abstract class AssetBundle(int assets) { - private int outstanding; - - internal void OnRequestCompleted(LightweightChannel ready) + protected readonly CountdownEvent Outstanding = new(assets); + internal void OnRequestCompleted() { - if (Interlocked.Decrement(ref outstanding) == 0) - { - ready.Write(this); - } + Outstanding.Signal(); } - - internal abstract void Materialize(); } public sealed class AssetBundle : AssetBundle + where T : notnull { - private readonly int Id; private readonly Func Resolver; - private readonly TaskCompletionSource Source = new(TaskCreationOptions.RunContinuationsAsynchronously); - internal AssetBundle(int id, Func resolver) + internal AssetBundle(int assets, Func resolver) + : base(assets) { - Id = id; Resolver = resolver; } - internal override void Materialize() + public T Wait(CancellationToken cancellationToken = default) { - Source.SetResult(Resolver(new PromiseResolver(Id))); + Outstanding.Wait(cancellationToken); + return Resolver(new PromiseResolver()); } - public Task Completion => Source.Task; + public bool Check([NotNullWhen(true)] out T? value) + { + if (Outstanding.IsSet) + { + value = Wait(); + return true; + } + value = default; + return false; + } } - public sealed record ExampleAssets(object A, string B) { public static AssetBundle Define(AssetManager assetManager) { - var transcoder = new ExampleTranscoder(); - - var a = assetManager.Load(new AssetId("key", "path"), transcoder, default); - var b = assetManager.Load(new AssetId("key", "path"), transcoder, default); + var a = assetManager.Load(new AssetId("key", "path"), default); + var b = assetManager.Load(new AssetId("key", "path"), default); return assetManager.Bundle(r => new ExampleAssets(r.Get(a), r.Get(b))); } - - public static async Task Foo(AssetBundle bundle) - { - ExampleAssets assets = await bundle.Completion; - } -} - - -public sealed class ExampleTranscoder() : NoSettingsTranscoder(Guid.NewGuid(), 1) -{ - public override Task Encode(AssetId id, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer) - { - writer.Write("Hello World"); - return Task.CompletedTask; - } - - public override string Decode(AssetId id, ref SequenceReader reader) - { - return reader.ReadString(); - } - - public override void HotSwap(string instance, string newParts) - { - throw new NotImplementedException(); - } } diff --git a/source/CapriKit.AssetPipeline/vNext/AssetManager.cs b/source/CapriKit.AssetPipeline/vNext/AssetManager.cs index 4a0d54c..057e7b4 100644 --- a/source/CapriKit.AssetPipeline/vNext/AssetManager.cs +++ b/source/CapriKit.AssetPipeline/vNext/AssetManager.cs @@ -1,27 +1,49 @@ -using CapriKit.Concurrency.Promises; +using CapriKit.Concurrency.Primitives; namespace CapriKit.AssetPipeline.vNext; public sealed partial class AssetManager : IDisposable { + // TODO: register and then get transcoders + private readonly Dictionary OutstandingPromises = []; + private readonly LightweightChannel<(AssetId Id, object Asset)> Ready = new(); + private readonly AssetCache Cache = new(); - - public bool TryLoad(AssetId id, IAssetTranscoder transcoder, TSetting settings) + public bool TryLoad(AssetId id, TSetting settings) where TAsset : class { + // TODO: Check if the file is in the AssetCache // TODO: Otherwise kick off the loading process // TODO: build to random output file and only move it to the right one once the main thread drains it. throw new NotImplementedException(); } - - public Promise Load(AssetId id, IAssetTranscoder transcoder, TSetting settings) + public Promise Load(AssetId id, TSetting settings) where TAsset : class { + var promise = new Promise(); + + throw new NotImplementedException(); } + public void Update() + { + // TODO: what if 2 promises wait for the same thing, multi-threading + // TODO: what if an asset is already done loading (or was already loaded) before the bundle was created + + while (Ready.TryRead(out var kv)) + { + if (OutstandingPromises.Remove(kv.Id, out var kv2)) + { + var asset = Cache.PutOrLease(kv.Id, kv.Asset); + kv2.Promise.Value = asset; + kv2.Bundle.OnRequestCompleted(); + } + } + } + public void Dispose() { @@ -34,6 +56,6 @@ internal AssetBundle Bundle(Func con // TODO: do something so that bundle gets notified when something is done loading // or maybe the actual loading should only start here? - return new AssetBundle(1, constructor); + return new AssetBundle(constructor); } } diff --git a/source/CapriKit.AssetPipeline/vNext/Promise.cs b/source/CapriKit.AssetPipeline/vNext/Promise.cs new file mode 100644 index 0000000..80b432a --- /dev/null +++ b/source/CapriKit.AssetPipeline/vNext/Promise.cs @@ -0,0 +1,27 @@ +namespace CapriKit.AssetPipeline.vNext; + +public abstract class Promise +{ + internal object? Value { get; set; } +} + +public sealed class Promise : Promise; + +public sealed class PromiseResolver +{ + public TValue Get(Promise promise) + { + // TODO: can we somehow double check that this resolver is able to resolve the promise? + //if (promise.Owner != Id) + //{ + // throw new InvalidOperationException($"Attempted to resolve a promise that was not owned by this resolver"); + //} + + if (promise.Value is TValue value) + { + return value; + } + + throw new Exception($"Internal error: resolved value was not of type {typeof(TValue).Name} but {promise.Value?.GetType().Name ?? "null"}"); + } +} diff --git a/source/CapriKit.Concurrency/Promises/Promise.cs b/source/CapriKit.Concurrency/Promises/Promise.cs deleted file mode 100644 index 3e0574d..0000000 --- a/source/CapriKit.Concurrency/Promises/Promise.cs +++ /dev/null @@ -1,34 +0,0 @@ -namespace CapriKit.Concurrency.Promises; - -public abstract class Promise -{ - internal Promise(int owner) - { - Owner = owner; - } - - internal int Owner { get; } - internal object? Value { get; set; } -} - -public sealed class Promise(int owner) : Promise(owner); - -public sealed class PromiseResolver(int id) -{ - private readonly int Id = id; - - public TValue Get(Promise promise) - { - if (promise.Owner != Id) - { - throw new InvalidOperationException($"Attempted to resolve a promise that was not owned by this resolver"); - } - - if (promise.Value is TValue value) - { - return value; - } - - throw new Exception($"Internal error: resolved value was not of type {typeof(TValue).Name} but {promise.Value?.GetType().Name ?? "null"}"); - } -} From 99c9cede963daacbbfed84f6b8400a91373c827a Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Tue, 11 Aug 2026 23:20:31 +0200 Subject: [PATCH 33/53] WIP: some progress --- .../vNext/AssetBundle.cs | 89 +++++++++++-------- .../vNext/AssetManager.cs | 41 ++------- .../CapriKit.AssetPipeline/vNext/Promise.cs | 27 ++++-- 3 files changed, 77 insertions(+), 80 deletions(-) diff --git a/source/CapriKit.AssetPipeline/vNext/AssetBundle.cs b/source/CapriKit.AssetPipeline/vNext/AssetBundle.cs index 865f2c9..a5ed2fe 100644 --- a/source/CapriKit.AssetPipeline/vNext/AssetBundle.cs +++ b/source/CapriKit.AssetPipeline/vNext/AssetBundle.cs @@ -1,64 +1,81 @@ -using CapriKit.IO; -using CapriKit.IO.Streams; -using System.Buffers; using System.Diagnostics.CodeAnalysis; namespace CapriKit.AssetPipeline.vNext; -// TODO: this and the promises in CapriKit.Concurrency implement the ideas from -// research\AssetPipelineLoadingGroups.md +// TODO: this implements the ideas from +// research\AssetPipelineLoadingGroupsV2.md // but there are still a few open questions -// - I need a blocking one so that all systems can start (Bootstrap), is that await? In CapriKit.Test.Tool I seem to be able to avoid that -// - I need a non-blocking one for all the other assets for stuff that should happen during loading screens what mechanism to use? -// - LightWeightChannel to the rescue and then swapping the entire 'loading scene' with the new scene? -// - When does the actual loading start and how do we register for it? See AssetManager.Bundle -// - How/when do we set the owner of the promise for the extra check? -public abstract class AssetBundle(int assets) -{ - protected readonly CountdownEvent Outstanding = new(assets); - internal void OnRequestCompleted() - { - Outstanding.Signal(); - } -} -public sealed class AssetBundle : AssetBundle - where T : notnull +public sealed class AssetBundleBuilder(AssetManager assetManager) { - private readonly Func Resolver; + private readonly List Promises = []; - internal AssetBundle(int assets, Func resolver) - : base(assets) + public Promise Load(AssetId id, TSettings settings) + where TAsset : class { - Resolver = resolver; + var promise = assetManager.Load(id, settings); + Promises.Add(promise); + return promise; } - public T Wait(CancellationToken cancellationToken = default) + public AssetBundle Build(Func factory) + where TBundle : notnull { - Outstanding.Wait(cancellationToken); - return Resolver(new PromiseResolver()); + var bundle = new AssetBundle(factory, Promises); + foreach (var promise in Promises) + { + promise.Owner = bundle; + } + + return bundle; } +} - public bool Check([NotNullWhen(true)] out T? value) +public abstract class AssetBundle +{ + +} + +public sealed class AssetBundle(Func factory, IReadOnlyList promises) + : AssetBundle + where TBundle : notnull +{ + private TBundle? result; + + // Single threaded! + // TODO: can we reduce the number of things we need to check each frame? + public bool IsReady([NotNullWhen(true)] out TBundle? value) { - if (Outstanding.IsSet) + if (result == null) { - value = Wait(); - return true; + foreach (var promise in promises) + { + if (!promise.IsResolved) + { + value = default; + return false; + } + } + + result = factory(new PromiseResolver(this)); } - value = default; - return false; + + value = result; + return true; } + + // TODO: how do we block and wait? } public sealed record ExampleAssets(object A, string B) { public static AssetBundle Define(AssetManager assetManager) { - var a = assetManager.Load(new AssetId("key", "path"), default); - var b = assetManager.Load(new AssetId("key", "path"), default); + var builder = new AssetBundleBuilder(assetManager); + var a = builder.Load(new AssetId("key", "path"), default); + var b = builder.Load(new AssetId("key", "path"), default); - return assetManager.Bundle(r => new ExampleAssets(r.Get(a), r.Get(b))); + return builder.Build(r => new ExampleAssets(r.Get(a), r.Get(b))); } } diff --git a/source/CapriKit.AssetPipeline/vNext/AssetManager.cs b/source/CapriKit.AssetPipeline/vNext/AssetManager.cs index 057e7b4..02b80a6 100644 --- a/source/CapriKit.AssetPipeline/vNext/AssetManager.cs +++ b/source/CapriKit.AssetPipeline/vNext/AssetManager.cs @@ -1,47 +1,25 @@ -using CapriKit.Concurrency.Primitives; - namespace CapriKit.AssetPipeline.vNext; public sealed partial class AssetManager : IDisposable { // TODO: register and then get transcoders - private readonly Dictionary OutstandingPromises = []; - private readonly LightweightChannel<(AssetId Id, object Asset)> Ready = new(); - private readonly AssetCache Cache = new(); - - public bool TryLoad(AssetId id, TSetting settings) - where TAsset : class - { - - // TODO: Check if the file is in the AssetCache - // TODO: Otherwise kick off the loading process - // TODO: build to random output file and only move it to the right one once the main thread drains it. - throw new NotImplementedException(); - } + private readonly AssetCache Cache = new(); public Promise Load(AssetId id, TSetting settings) where TAsset : class { var promise = new Promise(); + // TODO: capture the promise and set the .Value property as soon as the asset finishes loading + // like in the OnCompleted capture of Task.FireAndForget, also do something with failures. + // Ideally failures pop-up as soon as the promise is use to create the bundle throw new NotImplementedException(); } public void Update() { - // TODO: what if 2 promises wait for the same thing, multi-threading - // TODO: what if an asset is already done loading (or was already loaded) before the bundle was created - - while (Ready.TryRead(out var kv)) - { - if (OutstandingPromises.Remove(kv.Id, out var kv2)) - { - var asset = Cache.PutOrLease(kv.Id, kv.Asset); - kv2.Promise.Value = asset; - kv2.Bundle.OnRequestCompleted(); - } - } + } @@ -49,13 +27,4 @@ public void Dispose() { throw new NotImplementedException(); } - - internal AssetBundle Bundle(Func constructor) - { - throw new NotImplementedException(); - - // TODO: do something so that bundle gets notified when something is done loading - // or maybe the actual loading should only start here? - return new AssetBundle(constructor); - } } diff --git a/source/CapriKit.AssetPipeline/vNext/Promise.cs b/source/CapriKit.AssetPipeline/vNext/Promise.cs index 80b432a..968b9b6 100644 --- a/source/CapriKit.AssetPipeline/vNext/Promise.cs +++ b/source/CapriKit.AssetPipeline/vNext/Promise.cs @@ -1,21 +1,32 @@ +using System.Diagnostics; + namespace CapriKit.AssetPipeline.vNext; -public abstract class Promise +public abstract class Promise() { - internal object? Value { get; set; } + internal AssetBundle? Owner { get; set; } + internal bool IsResolved { get; private set; } + internal object? Value { get; private set; } + + internal void Resolve(object value) + { + Debug.Assert(IsResolved == false); + + Value = value; + IsResolved = true; + } } public sealed class Promise : Promise; -public sealed class PromiseResolver +public sealed class PromiseResolver(AssetBundle owner) { public TValue Get(Promise promise) { - // TODO: can we somehow double check that this resolver is able to resolve the promise? - //if (promise.Owner != Id) - //{ - // throw new InvalidOperationException($"Attempted to resolve a promise that was not owned by this resolver"); - //} + if (promise.Owner != owner) + { + throw new InvalidOperationException($"Attempted to resolve a promise that was not owned by the bundle"); + } if (promise.Value is TValue value) { From 4ebd417525d18c5fce8bfbf7bf083442817b7502 Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Wed, 12 Aug 2026 16:44:16 +0200 Subject: [PATCH 34/53] WIP: MOAR --- .../{AssetBundle.cs => AssetBundleLoader.cs} | 32 +-- .../vNext/AssetHandle.cs | 54 +++++ .../vNext/AssetManager.cs | 189 +++++++++++++++++- .../CapriKit.AssetPipeline/vNext/Promise.cs | 38 ---- 4 files changed, 248 insertions(+), 65 deletions(-) rename source/CapriKit.AssetPipeline/vNext/{AssetBundle.cs => AssetBundleLoader.cs} (56%) create mode 100644 source/CapriKit.AssetPipeline/vNext/AssetHandle.cs delete mode 100644 source/CapriKit.AssetPipeline/vNext/Promise.cs diff --git a/source/CapriKit.AssetPipeline/vNext/AssetBundle.cs b/source/CapriKit.AssetPipeline/vNext/AssetBundleLoader.cs similarity index 56% rename from source/CapriKit.AssetPipeline/vNext/AssetBundle.cs rename to source/CapriKit.AssetPipeline/vNext/AssetBundleLoader.cs index a5ed2fe..597293e 100644 --- a/source/CapriKit.AssetPipeline/vNext/AssetBundle.cs +++ b/source/CapriKit.AssetPipeline/vNext/AssetBundleLoader.cs @@ -9,36 +9,36 @@ namespace CapriKit.AssetPipeline.vNext; public sealed class AssetBundleBuilder(AssetManager assetManager) { - private readonly List Promises = []; + private readonly List Handles = []; - public Promise Load(AssetId id, TSettings settings) + public AssetHandle Load(AssetId id, TSettings settings) where TAsset : class { - var promise = assetManager.Load(id, settings); - Promises.Add(promise); - return promise; + var handle = assetManager.Load(id, settings); + Handles.Add(handle); + return handle; } - public AssetBundle Build(Func factory) + public AssetBundleLoader Build(Func factory) where TBundle : notnull { - var bundle = new AssetBundle(factory, Promises); - foreach (var promise in Promises) + var bundle = new AssetBundleLoader(factory, Handles); + foreach (var handle in Handles) { - promise.Owner = bundle; + handle.Owner = bundle; } return bundle; } } -public abstract class AssetBundle +public abstract class AssetBundleLoader { } -public sealed class AssetBundle(Func factory, IReadOnlyList promises) - : AssetBundle +public sealed class AssetBundleLoader(Func factory, IReadOnlyList handles) + : AssetBundleLoader where TBundle : notnull { private TBundle? result; @@ -49,16 +49,16 @@ public bool IsReady([NotNullWhen(true)] out TBundle? value) { if (result == null) { - foreach (var promise in promises) + foreach (var handle in handles) { - if (!promise.IsResolved) + if (!handle.IsResolved) { value = default; return false; } } - result = factory(new PromiseResolver(this)); + result = factory(new AssetHandleResolver(this)); } value = result; @@ -70,7 +70,7 @@ public bool IsReady([NotNullWhen(true)] out TBundle? value) public sealed record ExampleAssets(object A, string B) { - public static AssetBundle Define(AssetManager assetManager) + public static AssetBundleLoader Load(AssetManager assetManager) { var builder = new AssetBundleBuilder(assetManager); var a = builder.Load(new AssetId("key", "path"), default); diff --git a/source/CapriKit.AssetPipeline/vNext/AssetHandle.cs b/source/CapriKit.AssetPipeline/vNext/AssetHandle.cs new file mode 100644 index 0000000..51de36e --- /dev/null +++ b/source/CapriKit.AssetPipeline/vNext/AssetHandle.cs @@ -0,0 +1,54 @@ +using System.Diagnostics; + +namespace CapriKit.AssetPipeline.vNext; + +public abstract class AssetHandle() +{ + internal protected bool isResolved; + internal protected object? value; + + + internal AssetBundleLoader? Owner { get; set; } + internal bool IsResolved => isResolved; + internal object? Value => value; + + internal void Resolve(Exception exception) + { + Debug.Assert(isResolved == false); + value = exception; + isResolved = true; + } +} + +public sealed class AssetHandle : AssetHandle +{ + internal void Resolve(TValue asset) + { + Debug.Assert(isResolved == false); + value = asset; + isResolved = true; + } +} + +public sealed class AssetHandleResolver(AssetBundleLoader owner) +{ + public TValue Get(AssetHandle promise) + { + if (promise.Owner != owner) + { + throw new InvalidOperationException($"Attempted to resolve a promise that was not owned by the bundle"); + } + + if (promise.Value is TValue value) + { + return value; + } + + if (promise.Value is Exception ex) + { + throw ex; + } + + throw new Exception($"Internal error: resolved value was not of type {typeof(TValue).Name} but {promise.Value?.GetType().Name ?? "null"}"); + } +} diff --git a/source/CapriKit.AssetPipeline/vNext/AssetManager.cs b/source/CapriKit.AssetPipeline/vNext/AssetManager.cs index 02b80a6..d4d65d5 100644 --- a/source/CapriKit.AssetPipeline/vNext/AssetManager.cs +++ b/source/CapriKit.AssetPipeline/vNext/AssetManager.cs @@ -1,30 +1,197 @@ +using CapriKit.Concurrency.Async; +using CapriKit.IO; +using Microsoft.Extensions.Logging; +using System.Buffers; +using System.Collections.Concurrent; + namespace CapriKit.AssetPipeline.vNext; public sealed partial class AssetManager : IDisposable { - // TODO: register and then get transcoders - private readonly AssetCache Cache = new(); + // TODO: ensure that the transcoders and hot-reload manager are thread safe + // TODO: what if an asset is requested multiple times? Do we keep track + // of in-flight loading? Same for hot-reloading. + + private readonly ILogger Logger; + private readonly IVirtualFileSystem FileSystem; + private readonly AssetCache Cache; + private readonly HotReloadManager HotReloadManager; + private readonly ConcurrentDictionary Transcoders; + + public AssetManager(ILoggerFactory logger, ScopedFileSystem fileSystem) + { + Logger = logger.CreateLogger(); + FileSystem = fileSystem; + Cache = new AssetCache(); + HotReloadManager = new HotReloadManager(logger, fileSystem); + Transcoders = []; + } + + // Thread safe + public void RegisterTranscoder(IAssetTranscoder transcoder) + where TAsset : class + { + var key = typeof(TAsset); + var original = Transcoders.GetOrAdd(key, transcoder); + if (original != transcoder) + { + throw new Exception($"A transcoder for key: {key.FullName} was already registered"); + } + } - public Promise Load(AssetId id, TSetting settings) + // Thread safe + public AssetHandle Load(AssetId id, TSettings settings) where TAsset : class { - var promise = new Promise(); + var handle = new AssetHandle(); - // TODO: capture the promise and set the .Value property as soon as the asset finishes loading - // like in the OnCompleted capture of Task.FireAndForget, also do something with failures. - // Ideally failures pop-up as soon as the promise is use to create the bundle + // Check if the asset was loaded before + if (Cache.TryLease(id, out var cachedAsset)) + { + LogLoadedFromCache(Logger, id); + handle.Resolve(cachedAsset); + return handle; + } - throw new NotImplementedException(); + Task.Run(() => BuildAsset(id, settings, handle)).FireAndForget( + ex => + { + LogFailed(Logger, id); + handle.Resolve(ex); + }); + + return handle; } - public void Update() + // Thread safe is the AssetDecoder methods are thread safe + // TODO: take special care about the file that the asset is output to and when the copying of the file is resolved + // though that might be more of a thing for the hot reloader to worry about it can happen if the file + // is loaded twice. + // TODO: HIGH! If the same asset is requested multiple times at startup its build or loaded multiple times. + private async Task BuildAsset(AssetId id, TSettings settings, AssetHandle handle) + where TAsset : class { - + TAsset asset; + var transcoder = GetTranscoder(); + + // Check if the asset can be loaded from an up-to-date build + var build = await AssetDecoder.TryDecodeBuildMetaData(id, transcoder, FileSystem); + if (build != default && IsUpToDate(transcoder, settings, build, FileSystem)) + { + var upToDateAsset = await AssetDecoder.Decode(id, transcoder, FileSystem); + asset = RegisterAsset(upToDateAsset, transcoder); + handle.Resolve(asset); + + LogLoadedFromFile(Logger, id); + } + + // If not, try to rebuild and load the asset + if (!FileSystem.Exists(id.Path)) + { + throw new FileNotFoundException("Could not find primary file to build asset from", id.Path); + } + + await AssetEncoder.Encode(id, transcoder, settings, FileSystem); + var freshAsset = await AssetDecoder.Decode(id, transcoder, FileSystem); + asset = RegisterAsset(freshAsset, transcoder); + handle.Resolve(asset); + + LogBuildAndLoaded(Logger, id); } + // Thread safe + public void Unload(AssetId id) + { + Cache.Return(id); + } + + // Should only be called from the main thread, though technically thread safe + public void Update() + { + Cache.Collect(); + HotReloadManager.Update(); + } public void Dispose() { - throw new NotImplementedException(); + Cache.Dispose(); + HotReloadManager.Dispose(); } + + // Thread safe, only touches the file system and uses thread safe transcoder methods and properties. + private static bool IsUpToDate(IAssetTranscoder transcoder, TSettings settings, AssetBuildMetaData build, IReadOnlyVirtualFileSystem fileSystem) + where TAsset : class + { + // Transcoders differ + if (transcoder.Id != build.TranscoderId || transcoder.Version != build.TranscoderVersion) + { + return false; + } + + // Settings differ + var inUse = new ArrayBufferWriter(); + transcoder.WriteSettings(settings, inUse); + + var inFile = new ArrayBufferWriter(); + transcoder.WriteSettings(build.Settings, inFile); + + if (!inUse.WrittenSpan.SequenceEqual(inFile.WrittenSpan)) + { + return false; + } + + // Dependencies have changed + foreach (var (file, version) in build.Dependencies) + { + if (!fileSystem.Exists(file)) + { + return false; + } + + var lastWrite = fileSystem.LastWriteTime(file); + if (version != lastWrite) + { + return false; + } + } + + return true; + } + + // TODO: ensure that HotReloadManager.Track is thread safe + private TAsset RegisterAsset(Asset asset, IAssetTranscoder transcoder) + where TAsset : class + { + HotReloadManager.Track(asset, transcoder); + return Cache.PutOrLease(asset.Id, asset.Value); + } + + // Thread safe because Transcoders is thread safe + private IAssetTranscoder GetTranscoder() where TAsset : class + { + var type = typeof(TAsset); + if (!Transcoders.TryGetValue(type, out var transcoder)) + { + throw new Exception($"Missing transcoder for asset type: {type.FullName}"); + } + + if (transcoder is not IAssetTranscoder specializedTranscoder) + { + throw new Exception($"Expected transcoder of type: {type.FullName} but got transcoder of type: {transcoder.GetType().FullName}"); + } + + return specializedTranscoder; + } + + [LoggerMessage(Level = LogLevel.Information, Message = "Loaded asset: {asset} from cache")] + private static partial void LogLoadedFromCache(ILogger logger, AssetId asset); + + [LoggerMessage(Level = LogLevel.Information, Message = "Loaded up-to-date asset: {asset} from file")] + private static partial void LogLoadedFromFile(ILogger logger, AssetId asset); + + [LoggerMessage(Level = LogLevel.Information, Message = "Build and loaded fresh asset: {asset}")] + private static partial void LogBuildAndLoaded(ILogger logger, AssetId asset); + + [LoggerMessage(Level = LogLevel.Error, Message = "Building or loading asset: {asset} failed.")] + private static partial void LogFailed(ILogger logger, AssetId asset); } diff --git a/source/CapriKit.AssetPipeline/vNext/Promise.cs b/source/CapriKit.AssetPipeline/vNext/Promise.cs deleted file mode 100644 index 968b9b6..0000000 --- a/source/CapriKit.AssetPipeline/vNext/Promise.cs +++ /dev/null @@ -1,38 +0,0 @@ -using System.Diagnostics; - -namespace CapriKit.AssetPipeline.vNext; - -public abstract class Promise() -{ - internal AssetBundle? Owner { get; set; } - internal bool IsResolved { get; private set; } - internal object? Value { get; private set; } - - internal void Resolve(object value) - { - Debug.Assert(IsResolved == false); - - Value = value; - IsResolved = true; - } -} - -public sealed class Promise : Promise; - -public sealed class PromiseResolver(AssetBundle owner) -{ - public TValue Get(Promise promise) - { - if (promise.Owner != owner) - { - throw new InvalidOperationException($"Attempted to resolve a promise that was not owned by the bundle"); - } - - if (promise.Value is TValue value) - { - return value; - } - - throw new Exception($"Internal error: resolved value was not of type {typeof(TValue).Name} but {promise.Value?.GetType().Name ?? "null"}"); - } -} From e2ad0ce9290e8df0e56edf7f30c95d6226c5357b Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Thu, 13 Aug 2026 17:01:14 +0200 Subject: [PATCH 35/53] Try a channel approach and support requesting the same asset multiple times --- CapriKit.slnx | 4 +- .../vNext/AssetHandle.cs | 17 +--- .../vNext/AssetManager.cs | 79 +++++++++++++------ 3 files changed, 61 insertions(+), 39 deletions(-) diff --git a/CapriKit.slnx b/CapriKit.slnx index 7ec1e69..2851bfd 100644 --- a/CapriKit.slnx +++ b/CapriKit.slnx @@ -33,10 +33,10 @@ - + - + diff --git a/source/CapriKit.AssetPipeline/vNext/AssetHandle.cs b/source/CapriKit.AssetPipeline/vNext/AssetHandle.cs index 51de36e..a039e74 100644 --- a/source/CapriKit.AssetPipeline/vNext/AssetHandle.cs +++ b/source/CapriKit.AssetPipeline/vNext/AssetHandle.cs @@ -7,27 +7,21 @@ public abstract class AssetHandle() internal protected bool isResolved; internal protected object? value; - internal AssetBundleLoader? Owner { get; set; } internal bool IsResolved => isResolved; internal object? Value => value; - internal void Resolve(Exception exception) + internal void Resolve(object asset) { Debug.Assert(isResolved == false); - value = exception; + value = asset; isResolved = true; } } public sealed class AssetHandle : AssetHandle { - internal void Resolve(TValue asset) - { - Debug.Assert(isResolved == false); - value = asset; - isResolved = true; - } + } public sealed class AssetHandleResolver(AssetBundleLoader owner) @@ -44,11 +38,6 @@ public TValue Get(AssetHandle promise) return value; } - if (promise.Value is Exception ex) - { - throw ex; - } - throw new Exception($"Internal error: resolved value was not of type {typeof(TValue).Name} but {promise.Value?.GetType().Name ?? "null"}"); } } diff --git a/source/CapriKit.AssetPipeline/vNext/AssetManager.cs b/source/CapriKit.AssetPipeline/vNext/AssetManager.cs index d4d65d5..77ddd93 100644 --- a/source/CapriKit.AssetPipeline/vNext/AssetManager.cs +++ b/source/CapriKit.AssetPipeline/vNext/AssetManager.cs @@ -1,4 +1,5 @@ using CapriKit.Concurrency.Async; +using CapriKit.Concurrency.Primitives; using CapriKit.IO; using Microsoft.Extensions.Logging; using System.Buffers; @@ -13,18 +14,25 @@ public sealed partial class AssetManager : IDisposable // of in-flight loading? Same for hot-reloading. private readonly ILogger Logger; - private readonly IVirtualFileSystem FileSystem; + private readonly ScopedFileSystem FileSystem; private readonly AssetCache Cache; private readonly HotReloadManager HotReloadManager; private readonly ConcurrentDictionary Transcoders; + private readonly LightweightChannel<(AssetId, Func)> Incoming; + private readonly Lock RequestLock; + private readonly Dictionary> Outstanding; + public AssetManager(ILoggerFactory logger, ScopedFileSystem fileSystem) { Logger = logger.CreateLogger(); FileSystem = fileSystem; - Cache = new AssetCache(); - HotReloadManager = new HotReloadManager(logger, fileSystem); + Cache = new(); + HotReloadManager = new(logger, fileSystem); Transcoders = []; + Incoming = new(); + RequestLock = new(); + Outstanding = []; } // Thread safe @@ -45,20 +53,34 @@ public AssetHandle Load(AssetId id, TSettings setting { var handle = new AssetHandle(); - // Check if the asset was loaded before - if (Cache.TryLease(id, out var cachedAsset)) + // Ensure we can't miss an asset being done loading or being requested by another. + lock (RequestLock) { - LogLoadedFromCache(Logger, id); - handle.Resolve(cachedAsset); - return handle; - } + // Check if the asset was loaded before + if (Cache.TryLease(id, out var cachedAsset)) + { + LogLoadedFromCache(Logger, id); + handle.Resolve(cachedAsset); + return handle; + } - Task.Run(() => BuildAsset(id, settings, handle)).FireAndForget( + // Check if the asset was already requested + if (Outstanding.TryGetValue(id, out var requestors)) + { + requestors.Add(handle); + } + else + { + // Request the asset + Outstanding[id] = [handle]; + Task.Run(() => RequestAsset(id, settings)).FireAndForget( ex => { LogFailed(Logger, id); - handle.Resolve(ex); + Incoming.Write(ex); }); + } + } return handle; } @@ -67,11 +89,9 @@ public AssetHandle Load(AssetId id, TSettings setting // TODO: take special care about the file that the asset is output to and when the copying of the file is resolved // though that might be more of a thing for the hot reloader to worry about it can happen if the file // is loaded twice. - // TODO: HIGH! If the same asset is requested multiple times at startup its build or loaded multiple times. - private async Task BuildAsset(AssetId id, TSettings settings, AssetHandle handle) + private async Task RequestAsset(AssetId id, TSettings settings) where TAsset : class { - TAsset asset; var transcoder = GetTranscoder(); // Check if the asset can be loaded from an up-to-date build @@ -79,9 +99,7 @@ private async Task BuildAsset(AssetId id, TSettings settings, if (build != default && IsUpToDate(transcoder, settings, build, FileSystem)) { var upToDateAsset = await AssetDecoder.Decode(id, transcoder, FileSystem); - asset = RegisterAsset(upToDateAsset, transcoder); - handle.Resolve(asset); - + Incoming.Write((id, () => RegisterAsset(upToDateAsset, transcoder))); LogLoadedFromFile(Logger, id); } @@ -93,9 +111,7 @@ private async Task BuildAsset(AssetId id, TSettings settings, await AssetEncoder.Encode(id, transcoder, settings, FileSystem); var freshAsset = await AssetDecoder.Decode(id, transcoder, FileSystem); - asset = RegisterAsset(freshAsset, transcoder); - handle.Resolve(asset); - + Incoming.Write((id, () => RegisterAsset(freshAsset, transcoder))); LogBuildAndLoaded(Logger, id); } @@ -105,9 +121,24 @@ public void Unload(AssetId id) Cache.Return(id); } - // Should only be called from the main thread, though technically thread safe + // Should only be called from the primary thread, though technically thread safe public void Update() { + lock (RequestLock) + { + while (Incoming.TryRead(out var result)) + { + var (id, retriever) = result; + var handles = Outstanding[id]; + foreach (var handle in handles) + { + var asset = retriever(); + handle.Resolve(asset); + } + Outstanding.Remove(id); + } + } + Cache.Collect(); HotReloadManager.Update(); } @@ -162,8 +193,10 @@ private static bool IsUpToDate(IAssetTranscoder(Asset asset, IAssetTranscoder transcoder) where TAsset : class { - HotReloadManager.Track(asset, transcoder); - return Cache.PutOrLease(asset.Id, asset.Value); + var actualObject = Cache.PutOrLease(asset.Id, asset.Value); + var actualWrapper = new Asset(asset.Id, actualObject, asset.BuildMetaData); + HotReloadManager.Track(actualWrapper, transcoder); // TODO: track needs to be thread safe and ignore adding the same thing multiple times! + return actualObject; } // Thread safe because Transcoders is thread safe From 26c5dc26e2260621628f0eefba57516b846ba169 Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Thu, 13 Aug 2026 19:56:49 +0200 Subject: [PATCH 36/53] WIP: almost there --- .../Shaders/VertexShaderTranscoder.cs | 1 + source/CapriKit.AssetPipeline/AssetCache.cs | 94 ------------- source/CapriKit.AssetPipeline/AssetManager.cs | 128 ------------------ .../HotReloadManager.cs | 1 + .../CapriKit.AssetPipeline/HotReloadable.cs | 2 + .../{ => vNext}/Asset.cs | 0 .../vNext/AssetCache.cs | 13 +- .../{ => vNext}/AssetDecoder.cs | 6 +- .../{ => vNext}/AssetEncoder.cs | 8 +- .../vNext/AssetManager.cs | 67 ++++++--- .../{ => vNext}/AssetUtilities.cs | 2 +- .../{ => vNext}/IAssetTranscoder.cs | 4 +- .../{ => vNext}/NoSettingsTranscoder.cs | 2 +- 13 files changed, 69 insertions(+), 259 deletions(-) delete mode 100644 source/CapriKit.AssetPipeline/AssetCache.cs delete mode 100644 source/CapriKit.AssetPipeline/AssetManager.cs rename source/CapriKit.AssetPipeline/{ => vNext}/Asset.cs (100%) rename source/CapriKit.AssetPipeline/{ => vNext}/AssetDecoder.cs (93%) rename source/CapriKit.AssetPipeline/{ => vNext}/AssetEncoder.cs (88%) rename source/CapriKit.AssetPipeline/{ => vNext}/AssetUtilities.cs (89%) rename source/CapriKit.AssetPipeline/{ => vNext}/IAssetTranscoder.cs (92%) rename source/CapriKit.AssetPipeline/{ => vNext}/NoSettingsTranscoder.cs (94%) diff --git a/source/CapriKit.AssetPipeline.DirectX11/Shaders/VertexShaderTranscoder.cs b/source/CapriKit.AssetPipeline.DirectX11/Shaders/VertexShaderTranscoder.cs index f1824d0..658699d 100644 --- a/source/CapriKit.AssetPipeline.DirectX11/Shaders/VertexShaderTranscoder.cs +++ b/source/CapriKit.AssetPipeline.DirectX11/Shaders/VertexShaderTranscoder.cs @@ -1,3 +1,4 @@ +using CapriKit.AssetPipeline.vNext; using CapriKit.DirectX11; using CapriKit.DirectX11.Resources.Shaders; using CapriKit.IO; diff --git a/source/CapriKit.AssetPipeline/AssetCache.cs b/source/CapriKit.AssetPipeline/AssetCache.cs deleted file mode 100644 index de9ebc5..0000000 --- a/source/CapriKit.AssetPipeline/AssetCache.cs +++ /dev/null @@ -1,94 +0,0 @@ -using System.Diagnostics.CodeAnalysis; - -namespace CapriKit.AssetPipeline; - -/// -/// Simple cache that uses reference counting to decide when to clean-up a resource. Assets can be leased and returned at any time. -/// The actual disposing of objects only happens when the main thread calls . -/// -internal sealed class AssetCache : IDisposable -{ - private class Line(object asset, int refCount) - { - public object Asset { get; } = asset; - public int RefCount { get; set; } = refCount; - } - - private readonly Lock Lock = new(); - private readonly Dictionary Lines = []; - - public void Put(AssetId id, TAsset asset) - where TAsset : class - { - lock (Lock) - { - if (Lines.ContainsKey(id)) - { - throw new Exception($"Cache already contains asset: {id}."); - } - - var entry = new Line(asset, 1); - Lines.Add(id, entry); - } - } - - public bool TryLease(AssetId id, [NotNullWhen(true)] out TAsset? asset) - where TAsset : class - { - lock (Lock) - { - if (Lines.TryGetValue(id, out var entry)) - { - entry.RefCount = entry.RefCount + 1; - asset = (TAsset)entry.Asset; - return true; - } - } - - asset = default; - return false; - } - - public void Return(AssetId id) - { - lock (Lock) - { - var entry = Lines[id]; - entry.RefCount = entry.RefCount - 1; - } - } - - // TODO: can we dispose objects in the background or would that make the GPU unhappy when assets like textures are disposed at random times? - public void Collect() - { - List? toCollect = null; - lock (Lock) - { - foreach (var (key, value) in Lines) - { - if (value.RefCount <= 0) - { - toCollect = toCollect ?? []; - toCollect.Add(key); - } - } - - if (toCollect == null) { return; } - - foreach (var key in toCollect) - { - Lines.Remove(key, out var entry); - (entry?.Asset as IDisposable)?.Dispose(); - } - } - } - - public void Dispose() - { - foreach (var value in Lines.Values) - { - (value.Asset as IDisposable)?.Dispose(); - } - Lines.Clear(); - } -} diff --git a/source/CapriKit.AssetPipeline/AssetManager.cs b/source/CapriKit.AssetPipeline/AssetManager.cs deleted file mode 100644 index 71465a6..0000000 --- a/source/CapriKit.AssetPipeline/AssetManager.cs +++ /dev/null @@ -1,128 +0,0 @@ -using CapriKit.IO; -using Microsoft.Extensions.Logging; -using System.Buffers; - -namespace CapriKit.AssetPipeline; - -public sealed partial class AssetManager : IDisposable -{ - private readonly ILogger Logger; - private readonly IVirtualFileSystem FileSystem; - private readonly AssetCache Cache; - private readonly HotReloadManager HotReloadManager; - - public AssetManager(ILoggerFactory logger, ScopedFileSystem fileSystem) - { - Logger = logger.CreateLogger(); - FileSystem = fileSystem; - Cache = new AssetCache(); - HotReloadManager = new HotReloadManager(logger, fileSystem); - } - - public async Task Load(AssetId id, IAssetTranscoder transcoder, TSettings settings) - where TAsset : class - { - // Check if the asset was loaded before - if (Cache.TryLease(id, out var cachedAsset)) - { - LogLoadedFromCache(Logger, id); - return cachedAsset; - } - - // If not, check if it can be loaded from an up-to-date build - var build = await AssetDecoder.TryDecodeBuildMetaData(id, transcoder, FileSystem); - if (build != default && IsUpToDate(transcoder, settings, build)) - { - var upToDateAsset = await AssetDecoder.Decode(id, transcoder, FileSystem); - RegisterAsset(upToDateAsset, transcoder); - - LogLoadedFromFile(Logger, id); - return upToDateAsset.Value; - } - - // If not, try to rebuild and load the asset - if (!FileSystem.Exists(id.Path)) - { - throw new FileNotFoundException("Could not find primary file to build asset from", id.Path); - } - - await AssetEncoder.Encode(id, transcoder, settings, FileSystem); - var freshAsset = await AssetDecoder.Decode(id, transcoder, FileSystem); - RegisterAsset(freshAsset, transcoder); - - LogBuildAndLoaded(Logger, id); - return freshAsset.Value; - } - - public void Unload(AssetId id) - { - Cache.Return(id); - } - - /// Must be called from the main thread. - public void Update() - { - Cache.Collect(); - HotReloadManager.Update(); - } - - private bool IsUpToDate(IAssetTranscoder transcoder, TSettings settings, AssetBuildMetaData build) - where TAsset : class - { - // Transcoders differ - if (transcoder.Id != build.TranscoderId || transcoder.Version != build.TranscoderVersion) - { - return false; - } - - // Settings differ - var inUse = new ArrayBufferWriter(); - transcoder.WriteSettings(settings, inUse); - - var inFile = new ArrayBufferWriter(); - transcoder.WriteSettings(build.Settings, inFile); - - if (!inUse.WrittenSpan.SequenceEqual(inFile.WrittenSpan)) - { - return false; - } - - // Dependencies have changed - foreach (var (file, version) in build.Dependencies) - { - if (!FileSystem.Exists(file)) - { - return false; - } - - var lastWrite = FileSystem.LastWriteTime(file); - if (version != lastWrite) - { - return false; - } - } - - return true; - } - - private void RegisterAsset(Asset asset, IAssetTranscoder transcoder) - where TAsset : class - { - Cache.Put(asset.Id, asset.Value); - HotReloadManager.Track(asset, transcoder); - } - - public void Dispose() - { - Cache.Dispose(); - } - - [LoggerMessage(Level = LogLevel.Information, Message = "Loaded asset: {asset} from cache")] - private static partial void LogLoadedFromCache(ILogger logger, AssetId asset); - - [LoggerMessage(Level = LogLevel.Information, Message = "Loaded up-to-date asset: {asset} from file")] - private static partial void LogLoadedFromFile(ILogger logger, AssetId asset); - - [LoggerMessage(Level = LogLevel.Information, Message = "Build and loaded fresh asset: {asset}")] - private static partial void LogBuildAndLoaded(ILogger logger, AssetId asset); -} diff --git a/source/CapriKit.AssetPipeline/HotReloadManager.cs b/source/CapriKit.AssetPipeline/HotReloadManager.cs index aff1727..947b068 100644 --- a/source/CapriKit.AssetPipeline/HotReloadManager.cs +++ b/source/CapriKit.AssetPipeline/HotReloadManager.cs @@ -1,3 +1,4 @@ +using CapriKit.AssetPipeline.vNext; using CapriKit.Concurrency.Async; using CapriKit.IO; using CapriKit.IO.Watchers; diff --git a/source/CapriKit.AssetPipeline/HotReloadable.cs b/source/CapriKit.AssetPipeline/HotReloadable.cs index ab44d99..6aee521 100644 --- a/source/CapriKit.AssetPipeline/HotReloadable.cs +++ b/source/CapriKit.AssetPipeline/HotReloadable.cs @@ -1,3 +1,4 @@ +using CapriKit.AssetPipeline.vNext; using CapriKit.IO; using System.Collections.Concurrent; @@ -30,6 +31,7 @@ public override async Task Reload(IVirtualFileSystem fileSystem, ConcurrentQueue { if (!Instance.TryGetTarget(out var cold)) { return; } + // TODO: this should method should use a random input and output paths await AssetEncoder.Encode(Id, Transcoder, Settings, fileSystem); var hot = await AssetDecoder.Decode(Id, Transcoder, fileSystem); diff --git a/source/CapriKit.AssetPipeline/Asset.cs b/source/CapriKit.AssetPipeline/vNext/Asset.cs similarity index 100% rename from source/CapriKit.AssetPipeline/Asset.cs rename to source/CapriKit.AssetPipeline/vNext/Asset.cs diff --git a/source/CapriKit.AssetPipeline/vNext/AssetCache.cs b/source/CapriKit.AssetPipeline/vNext/AssetCache.cs index 9e4b64f..1a0bb32 100644 --- a/source/CapriKit.AssetPipeline/vNext/AssetCache.cs +++ b/source/CapriKit.AssetPipeline/vNext/AssetCache.cs @@ -23,7 +23,7 @@ private sealed class Entry(AssetId id, object asset, int refCount) /// /// Stores the given asset and then leases it. If another caller already stored the asset /// the is disposed and the stored instance is leased instead. - /// Thread-safe: loading the same asset twice is wasteful but harmless, after calling this + /// Threading, thread-safe: loading the same asset twice is wasteful but harmless, after calling this /// method users must stop referencing . /// public TAsset PutOrLease(AssetId id, TAsset candidate) @@ -48,7 +48,8 @@ public TAsset PutOrLease(AssetId id, TAsset candidate) } /// - /// Attempts to retrieve the asset with the given id. Thread-safe. + /// Attempts to retrieve the asset with the given id. + /// Threading: thread-safe. /// public bool TryLease(AssetId id, [NotNullWhen(true)] out TAsset? asset) where TAsset : class @@ -70,9 +71,9 @@ public bool TryLease(AssetId id, [NotNullWhen(true)] out TAsset? asset) } /// - /// Returns a leased asset. If every user returned their asset it become collectable. Which happens in + /// Returns a leased asset. If every user returned their asset it becomes collectable. Which happens in /// . After calling return the caller must no longer reference the asset instance. - /// Thread-safe. + /// Threading: thread-safe. /// public void Return(AssetId id) { @@ -98,8 +99,8 @@ public void Return(AssetId id) } /// - /// Disposes all assets that no longer have users. This method must be called from the primary thread, - /// but it is safe for threads to concurrently access other methods in this class. + /// Disposes all assets that no longer have users. + /// Threading: this method must only be called from the primary thread. /// public void Collect() { diff --git a/source/CapriKit.AssetPipeline/AssetDecoder.cs b/source/CapriKit.AssetPipeline/vNext/AssetDecoder.cs similarity index 93% rename from source/CapriKit.AssetPipeline/AssetDecoder.cs rename to source/CapriKit.AssetPipeline/vNext/AssetDecoder.cs index 4ad7369..85d87ab 100644 --- a/source/CapriKit.AssetPipeline/AssetDecoder.cs +++ b/source/CapriKit.AssetPipeline/vNext/AssetDecoder.cs @@ -1,13 +1,15 @@ using CapriKit.IO; using CapriKit.IO.Streams; using System.Buffers; -using static CapriKit.AssetPipeline.AssetUtilities; +using static CapriKit.AssetPipeline.vNext.AssetUtilities; -namespace CapriKit.AssetPipeline; +namespace CapriKit.AssetPipeline.vNext; /// /// Decodes the generic asset envelope and, using a specialized IAssetTranscoder, the asset itself +/// Threading: thread-safe /// +// TODO: AssetDecoder it should be possible to override the output path internal static class AssetDecoder { public static async Task> Decode(AssetId id, IAssetTranscoder decoder, IReadOnlyVirtualFileSystem fileSystem) diff --git a/source/CapriKit.AssetPipeline/AssetEncoder.cs b/source/CapriKit.AssetPipeline/vNext/AssetEncoder.cs similarity index 88% rename from source/CapriKit.AssetPipeline/AssetEncoder.cs rename to source/CapriKit.AssetPipeline/vNext/AssetEncoder.cs index 2b5542b..8a141f4 100644 --- a/source/CapriKit.AssetPipeline/AssetEncoder.cs +++ b/source/CapriKit.AssetPipeline/vNext/AssetEncoder.cs @@ -2,15 +2,17 @@ using CapriKit.IO.Streams; using System.Buffers; using System.IO.Pipelines; -using static CapriKit.AssetPipeline.AssetUtilities; +using static CapriKit.AssetPipeline.vNext.AssetUtilities; -namespace CapriKit.AssetPipeline; +namespace CapriKit.AssetPipeline.vNext; // File format: [encoder id][encoder version][settings length][settings][payload length][payload][dependency count][dependencies] /// -/// Encodes the generic asset envelope and, using a specialized IAssetTranscoder, the asset itself +/// Encodes the generic asset envelope and, using a specialized IAssetTranscoder, the asset itself. +/// Threading: thread-safe /// +// TODO: AssetEncoder it should be possible to override the output path internal static class AssetEncoder { public static async Task Encode(AssetId id, IAssetTranscoder encoder, TSettings settings, IVirtualFileSystem fileSystem) diff --git a/source/CapriKit.AssetPipeline/vNext/AssetManager.cs b/source/CapriKit.AssetPipeline/vNext/AssetManager.cs index 77ddd93..4bbf79e 100644 --- a/source/CapriKit.AssetPipeline/vNext/AssetManager.cs +++ b/source/CapriKit.AssetPipeline/vNext/AssetManager.cs @@ -19,7 +19,7 @@ public sealed partial class AssetManager : IDisposable private readonly HotReloadManager HotReloadManager; private readonly ConcurrentDictionary Transcoders; - private readonly LightweightChannel<(AssetId, Func)> Incoming; + private readonly LightweightChannel<(AssetId Id, Func Materializer)> Incoming; private readonly Lock RequestLock; private readonly Dictionary> Outstanding; @@ -35,7 +35,11 @@ public AssetManager(ILoggerFactory logger, ScopedFileSystem fileSystem) Outstanding = []; } - // Thread safe + /// + /// Register a transcoder for the given asset type. Registering a transcoders for a type + /// that was already assigned a transcoder throws an exception. + /// Threading: thread-safe, multiple threads can register transcoders at the same time. + /// public void RegisterTranscoder(IAssetTranscoder transcoder) where TAsset : class { @@ -47,13 +51,20 @@ public void RegisterTranscoder(IAssetTranscoder + /// Starts loading an asset. The asset will either be loaded from the cache, from disk, or rebuild and then loaded. + /// The caller gets a handle to be used in an which can be resolved + /// to the actual asset when loading finishes using + /// Threading: thread-safe, can be called from any thread concurrently. This method guarantees that the same asset + /// is not loaded multiple times concurrently. + /// public AssetHandle Load(AssetId id, TSettings settings) where TAsset : class { var handle = new AssetHandle(); - // Ensure we can't miss an asset being done loading or being requested by another. + // At this time an asset is either already loaded, already requested or requested for the first time. + // The lock ensure that this does not change while we check what we should do with the request. lock (RequestLock) { // Check if the asset was loaded before @@ -85,10 +96,10 @@ public AssetHandle Load(AssetId id, TSettings setting return handle; } - // Thread safe is the AssetDecoder methods are thread safe - // TODO: take special care about the file that the asset is output to and when the copying of the file is resolved - // though that might be more of a thing for the hot reloader to worry about it can happen if the file - // is loaded twice. + /// + /// Performs the actual loading or building and loading of the asset. + /// Threading: The caller has to guarantee that this method does not run concurrently for the same asset-id. + /// private async Task RequestAsset(AssetId id, TSettings settings) where TAsset : class { @@ -99,7 +110,7 @@ private async Task RequestAsset(AssetId id, TSettings setting if (build != default && IsUpToDate(transcoder, settings, build, FileSystem)) { var upToDateAsset = await AssetDecoder.Decode(id, transcoder, FileSystem); - Incoming.Write((id, () => RegisterAsset(upToDateAsset, transcoder))); + Incoming.Write((id, () => MaterializeAsset(upToDateAsset, transcoder))); LogLoadedFromFile(Logger, id); } @@ -111,7 +122,7 @@ private async Task RequestAsset(AssetId id, TSettings setting await AssetEncoder.Encode(id, transcoder, settings, FileSystem); var freshAsset = await AssetDecoder.Decode(id, transcoder, FileSystem); - Incoming.Write((id, () => RegisterAsset(freshAsset, transcoder))); + Incoming.Write((id, () => MaterializeAsset(freshAsset, transcoder))); LogBuildAndLoaded(Logger, id); } @@ -121,18 +132,24 @@ public void Unload(AssetId id) Cache.Return(id); } - // Should only be called from the primary thread, though technically thread safe + /// + /// Materializes assets that have finished loading, removes unused items from the cache + /// and hot-reloads changed assets. + /// Threading: Should only be called from the primary thread. + /// public void Update() { + // Ensure that while we check which assets are done loading and up-to-date that administration + // a new request to `Load` of the same asset does not miss these update. lock (RequestLock) { while (Incoming.TryRead(out var result)) { - var (id, retriever) = result; + var (id, materializer) = result; var handles = Outstanding[id]; foreach (var handle in handles) { - var asset = retriever(); + var asset = materializer(); handle.Resolve(asset); } Outstanding.Remove(id); @@ -174,15 +191,15 @@ private static bool IsUpToDate(IAssetTranscoder(IAssetTranscoder(Asset asset, IAssetTranscoder transcoder) + /// + /// Used when an asset is loaded and the handler resolved to register the asset with the cache and hot-reloader. + /// Thread safe: calling this method from multiple threads, even to materialize the same asset is safe. + /// + private TAsset MaterializeAsset(Asset asset, IAssetTranscoder transcoder) where TAsset : class { + // Though the asset manager does not allow loading the same asset multiple times the cache contract does allow it var actualObject = Cache.PutOrLease(asset.Id, asset.Value); + var actualWrapper = new Asset(asset.Id, actualObject, asset.BuildMetaData); HotReloadManager.Track(actualWrapper, transcoder); // TODO: track needs to be thread safe and ignore adding the same thing multiple times! return actualObject; diff --git a/source/CapriKit.AssetPipeline/AssetUtilities.cs b/source/CapriKit.AssetPipeline/vNext/AssetUtilities.cs similarity index 89% rename from source/CapriKit.AssetPipeline/AssetUtilities.cs rename to source/CapriKit.AssetPipeline/vNext/AssetUtilities.cs index 1060446..b07cfbb 100644 --- a/source/CapriKit.AssetPipeline/AssetUtilities.cs +++ b/source/CapriKit.AssetPipeline/vNext/AssetUtilities.cs @@ -1,6 +1,6 @@ using CapriKit.IO; -namespace CapriKit.AssetPipeline; +namespace CapriKit.AssetPipeline.vNext; internal static class AssetUtilities { diff --git a/source/CapriKit.AssetPipeline/IAssetTranscoder.cs b/source/CapriKit.AssetPipeline/vNext/IAssetTranscoder.cs similarity index 92% rename from source/CapriKit.AssetPipeline/IAssetTranscoder.cs rename to source/CapriKit.AssetPipeline/vNext/IAssetTranscoder.cs index 51a7a68..29859f1 100644 --- a/source/CapriKit.AssetPipeline/IAssetTranscoder.cs +++ b/source/CapriKit.AssetPipeline/vNext/IAssetTranscoder.cs @@ -1,11 +1,11 @@ using CapriKit.IO; using System.Buffers; -namespace CapriKit.AssetPipeline; +namespace CapriKit.AssetPipeline.vNext; /// /// Interface for classes that builds assets (such as texture, models and sound effects) and load them -/// when the program needs them. +/// when the program needs them. Implementers must ensure that all methods are thread-safe. /// public interface IAssetTranscoder { diff --git a/source/CapriKit.AssetPipeline/NoSettingsTranscoder.cs b/source/CapriKit.AssetPipeline/vNext/NoSettingsTranscoder.cs similarity index 94% rename from source/CapriKit.AssetPipeline/NoSettingsTranscoder.cs rename to source/CapriKit.AssetPipeline/vNext/NoSettingsTranscoder.cs index a86ec8c..0810a2a 100644 --- a/source/CapriKit.AssetPipeline/NoSettingsTranscoder.cs +++ b/source/CapriKit.AssetPipeline/vNext/NoSettingsTranscoder.cs @@ -1,7 +1,7 @@ using CapriKit.IO; using System.Buffers; -namespace CapriKit.AssetPipeline; +namespace CapriKit.AssetPipeline.vNext; public readonly struct NoSettings; From c0bf7f81c105ed60a9f3384ce0f9854f9f17ca32 Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Thu, 13 Aug 2026 23:06:02 +0200 Subject: [PATCH 37/53] Ready for review --- .../{vNext => }/Asset.cs | 0 .../{vNext => }/AssetBundleLoader.cs | 7 +- .../{vNext => }/AssetCache.cs | 2 +- .../{vNext => }/AssetDecoder.cs | 8 +- .../{vNext => }/AssetEncoder.cs | 8 +- .../{vNext => }/AssetHandle.cs | 7 +- .../{vNext => }/AssetManager.cs | 2 +- .../{vNext => }/AssetUtilities.cs | 2 +- .../HotReloadManager.cs | 104 ++++++++++++++---- .../CapriKit.AssetPipeline/HotReloadable.cs | 15 ++- .../{vNext => }/IAssetTranscoder.cs | 2 +- .../{vNext => }/NoSettingsTranscoder.cs | 2 +- 12 files changed, 111 insertions(+), 48 deletions(-) rename source/CapriKit.AssetPipeline/{vNext => }/Asset.cs (100%) rename source/CapriKit.AssetPipeline/{vNext => }/AssetBundleLoader.cs (89%) rename source/CapriKit.AssetPipeline/{vNext => }/AssetCache.cs (96%) rename source/CapriKit.AssetPipeline/{vNext => }/AssetDecoder.cs (92%) rename source/CapriKit.AssetPipeline/{vNext => }/AssetEncoder.cs (88%) rename source/CapriKit.AssetPipeline/{vNext => }/AssetHandle.cs (87%) rename source/CapriKit.AssetPipeline/{vNext => }/AssetManager.cs (97%) rename source/CapriKit.AssetPipeline/{vNext => }/AssetUtilities.cs (89%) rename source/CapriKit.AssetPipeline/{vNext => }/IAssetTranscoder.cs (96%) rename source/CapriKit.AssetPipeline/{vNext => }/NoSettingsTranscoder.cs (94%) diff --git a/source/CapriKit.AssetPipeline/vNext/Asset.cs b/source/CapriKit.AssetPipeline/Asset.cs similarity index 100% rename from source/CapriKit.AssetPipeline/vNext/Asset.cs rename to source/CapriKit.AssetPipeline/Asset.cs diff --git a/source/CapriKit.AssetPipeline/vNext/AssetBundleLoader.cs b/source/CapriKit.AssetPipeline/AssetBundleLoader.cs similarity index 89% rename from source/CapriKit.AssetPipeline/vNext/AssetBundleLoader.cs rename to source/CapriKit.AssetPipeline/AssetBundleLoader.cs index 597293e..8b94219 100644 --- a/source/CapriKit.AssetPipeline/vNext/AssetBundleLoader.cs +++ b/source/CapriKit.AssetPipeline/AssetBundleLoader.cs @@ -1,11 +1,6 @@ using System.Diagnostics.CodeAnalysis; -namespace CapriKit.AssetPipeline.vNext; - -// TODO: this implements the ideas from -// research\AssetPipelineLoadingGroupsV2.md -// but there are still a few open questions - +namespace CapriKit.AssetPipeline; public sealed class AssetBundleBuilder(AssetManager assetManager) { diff --git a/source/CapriKit.AssetPipeline/vNext/AssetCache.cs b/source/CapriKit.AssetPipeline/AssetCache.cs similarity index 96% rename from source/CapriKit.AssetPipeline/vNext/AssetCache.cs rename to source/CapriKit.AssetPipeline/AssetCache.cs index 1a0bb32..4a8fd32 100644 --- a/source/CapriKit.AssetPipeline/vNext/AssetCache.cs +++ b/source/CapriKit.AssetPipeline/AssetCache.cs @@ -1,6 +1,6 @@ using System.Diagnostics.CodeAnalysis; -namespace CapriKit.AssetPipeline.vNext; +namespace CapriKit.AssetPipeline; /// /// Cache of live assets. Methods are thread-safe and can be accessed concurrently. However, diff --git a/source/CapriKit.AssetPipeline/vNext/AssetDecoder.cs b/source/CapriKit.AssetPipeline/AssetDecoder.cs similarity index 92% rename from source/CapriKit.AssetPipeline/vNext/AssetDecoder.cs rename to source/CapriKit.AssetPipeline/AssetDecoder.cs index 85d87ab..2222bc6 100644 --- a/source/CapriKit.AssetPipeline/vNext/AssetDecoder.cs +++ b/source/CapriKit.AssetPipeline/AssetDecoder.cs @@ -1,9 +1,9 @@ using CapriKit.IO; using CapriKit.IO.Streams; using System.Buffers; -using static CapriKit.AssetPipeline.vNext.AssetUtilities; +using static CapriKit.AssetPipeline.AssetUtilities; -namespace CapriKit.AssetPipeline.vNext; +namespace CapriKit.AssetPipeline; /// /// Decodes the generic asset envelope and, using a specialized IAssetTranscoder, the asset itself @@ -12,7 +12,7 @@ namespace CapriKit.AssetPipeline.vNext; // TODO: AssetDecoder it should be possible to override the output path internal static class AssetDecoder { - public static async Task> Decode(AssetId id, IAssetTranscoder decoder, IReadOnlyVirtualFileSystem fileSystem) + public static async Task> Decode(AssetId id, IAssetTranscoder decoder, IReadOnlyVirtualFileSystem fileSystem, Stream? inputStreamOverride = default) where TAsset : class { var inputPath = ToEncodedFilePath(id); @@ -21,7 +21,7 @@ public static async Task> Decode(Ass throw new FileNotFoundException($"Could not find file: {inputPath} to load asset: {id}", id.Path); } - using var input = fileSystem.OpenRead(inputPath); + using var input = inputStreamOverride ?? fileSystem.OpenRead(inputPath); var length = checked((int)input.Length); var buffer = ArrayPool.Shared.Rent(length); diff --git a/source/CapriKit.AssetPipeline/vNext/AssetEncoder.cs b/source/CapriKit.AssetPipeline/AssetEncoder.cs similarity index 88% rename from source/CapriKit.AssetPipeline/vNext/AssetEncoder.cs rename to source/CapriKit.AssetPipeline/AssetEncoder.cs index 8a141f4..44c73aa 100644 --- a/source/CapriKit.AssetPipeline/vNext/AssetEncoder.cs +++ b/source/CapriKit.AssetPipeline/AssetEncoder.cs @@ -2,9 +2,9 @@ using CapriKit.IO.Streams; using System.Buffers; using System.IO.Pipelines; -using static CapriKit.AssetPipeline.vNext.AssetUtilities; +using static CapriKit.AssetPipeline.AssetUtilities; -namespace CapriKit.AssetPipeline.vNext; +namespace CapriKit.AssetPipeline; // File format: [encoder id][encoder version][settings length][settings][payload length][payload][dependency count][dependencies] @@ -15,13 +15,13 @@ namespace CapriKit.AssetPipeline.vNext; // TODO: AssetEncoder it should be possible to override the output path internal static class AssetEncoder { - public static async Task Encode(AssetId id, IAssetTranscoder encoder, TSettings settings, IVirtualFileSystem fileSystem) + public static async Task Encode(AssetId id, IAssetTranscoder encoder, TSettings settings, IVirtualFileSystem fileSystem, Stream? outputStreamOverride = default) where TAsset : class { ThrowOnFileNotFound(id.Path, fileSystem); var outputPath = ToEncodedFilePath(id); - using var output = fileSystem.CreateReadWrite(outputPath); + using var output = outputStreamOverride ?? fileSystem.CreateReadWrite(outputPath); var writer = PipeWriter.Create(output); var spy = fileSystem.SpyOn(); diff --git a/source/CapriKit.AssetPipeline/vNext/AssetHandle.cs b/source/CapriKit.AssetPipeline/AssetHandle.cs similarity index 87% rename from source/CapriKit.AssetPipeline/vNext/AssetHandle.cs rename to source/CapriKit.AssetPipeline/AssetHandle.cs index a039e74..767dbc2 100644 --- a/source/CapriKit.AssetPipeline/vNext/AssetHandle.cs +++ b/source/CapriKit.AssetPipeline/AssetHandle.cs @@ -1,6 +1,6 @@ using System.Diagnostics; -namespace CapriKit.AssetPipeline.vNext; +namespace CapriKit.AssetPipeline; public abstract class AssetHandle() { @@ -19,10 +19,7 @@ internal void Resolve(object asset) } } -public sealed class AssetHandle : AssetHandle -{ - -} +public sealed class AssetHandle : AssetHandle { } public sealed class AssetHandleResolver(AssetBundleLoader owner) { diff --git a/source/CapriKit.AssetPipeline/vNext/AssetManager.cs b/source/CapriKit.AssetPipeline/AssetManager.cs similarity index 97% rename from source/CapriKit.AssetPipeline/vNext/AssetManager.cs rename to source/CapriKit.AssetPipeline/AssetManager.cs index 4bbf79e..436dac4 100644 --- a/source/CapriKit.AssetPipeline/vNext/AssetManager.cs +++ b/source/CapriKit.AssetPipeline/AssetManager.cs @@ -5,7 +5,7 @@ using System.Buffers; using System.Collections.Concurrent; -namespace CapriKit.AssetPipeline.vNext; +namespace CapriKit.AssetPipeline; public sealed partial class AssetManager : IDisposable { diff --git a/source/CapriKit.AssetPipeline/vNext/AssetUtilities.cs b/source/CapriKit.AssetPipeline/AssetUtilities.cs similarity index 89% rename from source/CapriKit.AssetPipeline/vNext/AssetUtilities.cs rename to source/CapriKit.AssetPipeline/AssetUtilities.cs index b07cfbb..1060446 100644 --- a/source/CapriKit.AssetPipeline/vNext/AssetUtilities.cs +++ b/source/CapriKit.AssetPipeline/AssetUtilities.cs @@ -1,6 +1,6 @@ using CapriKit.IO; -namespace CapriKit.AssetPipeline.vNext; +namespace CapriKit.AssetPipeline; internal static class AssetUtilities { diff --git a/source/CapriKit.AssetPipeline/HotReloadManager.cs b/source/CapriKit.AssetPipeline/HotReloadManager.cs index 947b068..c6002d9 100644 --- a/source/CapriKit.AssetPipeline/HotReloadManager.cs +++ b/source/CapriKit.AssetPipeline/HotReloadManager.cs @@ -1,4 +1,3 @@ -using CapriKit.AssetPipeline.vNext; using CapriKit.Concurrency.Async; using CapriKit.IO; using CapriKit.IO.Watchers; @@ -22,7 +21,8 @@ internal sealed partial class HotReloadManager : IDisposable private readonly IVirtualFileSystemWatcher Watcher; private readonly FileSystemEventQueue FileChances; - private readonly Dictionary Tracked; + private readonly Lock TrackingLock; + private readonly Dictionary> Tracked; private readonly Dictionary> Dependents; private readonly HashSet PendingRebuilds; @@ -37,8 +37,9 @@ public HotReloadManager(ILoggerFactory logger, ScopedFileSystem fileSystem) FileSystem = fileSystem; Watcher = fileSystem.Watch(); - FileChances = new FileSystemEventQueue(Watcher); + FileChances = new(Watcher); + TrackingLock = new(); Tracked = []; Dependents = []; @@ -49,25 +50,49 @@ public HotReloadManager(ILoggerFactory logger, ScopedFileSystem fileSystem) isReloading = false; } + /// + /// Registers an asset for tracking by the hot-reload system. + /// Threading: thread-safe, multiple threads can call this method and can even register + /// the same asset multiple times. This class figures out which instances are still relevant. + /// public void Track(Asset asset, IAssetTranscoder transcoder) where TAsset : class { - Tracked[asset.Id] = new HotReloadable(asset, transcoder); - foreach (var dependency in asset.BuildMetaData.Dependencies) + // An asset can be registered multiple times + lock (TrackingLock) { - var file = dependency.File; - if (Dependents.TryGetValue(file, out var ids)) + var reloadable = new HotReloadable(asset, transcoder); + if (Tracked.TryGetValue(asset.Id, out var assets)) { - ids.Add(asset.Id); + // Prevent adding the exact same instance multiple times + if (assets.Any(a => object.ReferenceEquals(a, asset))) { return; } + assets.Add(reloadable); } else { - ids = [asset.Id]; - Dependents.Add(file, ids); + Tracked[asset.Id] = [reloadable]; + } + + foreach (var dependency in asset.BuildMetaData.Dependencies) + { + var file = dependency.File; + if (Dependents.TryGetValue(file, out var ids)) + { + ids.Add(asset.Id); + } + else + { + ids = [asset.Id]; + Dependents.Add(file, ids); + } } } } + /// + /// Checks which files required reloading, start the reloading process and hot swaps any asset that have reloaded + /// Threading: Unsafe, must only be called by the primary thread. Other threads can call other methods in this class. + /// public void Update() { if (isReloading) @@ -85,6 +110,11 @@ public void Update() HotSwapPending(); } + /// + /// Drains the queue of file events and adds any assets that dependent on this file to PendingRebuilds + /// Threading: Unsafe, must be called single threaded because the Dependents dictionary can only + /// be used by one thread at a time. + /// private void DrainFileChanges() { while (FileChances.TryDequeue(out var @event)) @@ -101,19 +131,48 @@ private void DrainFileChanges() } } + /// + /// Starts rebuilding the first asset in the set + /// Threading: Unsafe, must be called single threaded because the PendingRebuilds sets can only + /// be used by one thread at a time and the isReloading guard would also be confused. + /// private void ReloadOne() { - if (PendingRebuilds.Count > 0) - { - var id = PendingRebuilds.First(); - PendingRebuilds.Remove(id); + if (PendingRebuilds.Count == 0) { return; } + + var id = PendingRebuilds.First(); + PendingRebuilds.Remove(id); + + isReloading = true; + + LogReloadStarted(Logger, id); - isReloading = true; + HotReloadable? target = null; + List candidates; - LogReloadStarted(Logger, id); + // Even though this method is single threaded, other methods that allow parallelism can + // touch Tracked so we need to put a lock around it. + lock (TrackingLock) + { + candidates = Tracked[id]; + } - var reloadable = Tracked[id]; - reloadable.Reload(FileSystem, PendingReloads) + if (candidates.Count > 0) + { + var pruned = new List(1); + foreach (var candidate in candidates) + { + if (candidate.IsAlive) + { + pruned.Add(candidate); + target = candidate; + } + } + } + + // Not finding a target is normal, it means a file + // changed but the asset depending on it is no longer in use. + target?.Reload(FileSystem, PendingReloads) .FireAndForget(ex => { LogReloadFailed(Logger, id, ex); @@ -124,9 +183,14 @@ private void ReloadOne() LogReloadCompleted(Logger, id); isReloading = false; }); - } + } + /// + /// Hot swaps the assets that have been reloaded. + /// Threading: Unsafe, the contract from used here + /// requires that assets are only hot swapped on the main thread + /// private void HotSwapPending() { while (PendingReloads.TryDequeue(out var action)) @@ -156,7 +220,7 @@ public void Dispose() [LoggerMessage(Level = LogLevel.Information, Message = "Detected file change: {path}, affecting asset: {asset}")] private static partial void LogPendingReload(ILogger logger, FilePath path, AssetId asset); - [LoggerMessage(Level = LogLevel.Information, Message = "Reloading asset started: {asset}")] + [LoggerMessage(Level = LogLevel.Information, Message = "Reloading asset pending: {asset}")] private static partial void LogReloadStarted(ILogger logger, AssetId asset); [LoggerMessage(Level = LogLevel.Error, Message = "Reloading asset completed: {asset}")] diff --git a/source/CapriKit.AssetPipeline/HotReloadable.cs b/source/CapriKit.AssetPipeline/HotReloadable.cs index 6aee521..14891b3 100644 --- a/source/CapriKit.AssetPipeline/HotReloadable.cs +++ b/source/CapriKit.AssetPipeline/HotReloadable.cs @@ -1,4 +1,3 @@ -using CapriKit.AssetPipeline.vNext; using CapriKit.IO; using System.Collections.Concurrent; @@ -9,6 +8,8 @@ internal sealed record HotSwapAction(AssetId Id, Action PerformHotSwap); internal abstract class HotReloadable(AssetId id) { public AssetId Id { get; } = id; + public abstract bool IsAlive { get; } + public abstract Task Reload(IVirtualFileSystem fileSystem, ConcurrentQueue hotSwapActionQueue); } @@ -27,13 +28,19 @@ public HotReloadable(Asset asset, IAssetTranscoder Instance.TryGetTarget(out var _); + public override async Task Reload(IVirtualFileSystem fileSystem, ConcurrentQueue hotSwapActionQueue) { if (!Instance.TryGetTarget(out var cold)) { return; } - // TODO: this should method should use a random input and output paths - await AssetEncoder.Encode(Id, Transcoder, Settings, fileSystem); - var hot = await AssetDecoder.Decode(Id, Transcoder, fileSystem); + // We store the encoded asset in memory instead of on disk to prevent + // touching the file while other threads are also working on it. + using var stream = new MemoryStream(); + await AssetEncoder.Encode(Id, Transcoder, Settings, fileSystem, stream); + + stream.Seek(0, SeekOrigin.Begin); + var hot = await AssetDecoder.Decode(Id, Transcoder, fileSystem, stream); hotSwapActionQueue.Enqueue(new HotSwapAction(Id, () => Transcoder.HotSwap(cold, hot.Value))); } diff --git a/source/CapriKit.AssetPipeline/vNext/IAssetTranscoder.cs b/source/CapriKit.AssetPipeline/IAssetTranscoder.cs similarity index 96% rename from source/CapriKit.AssetPipeline/vNext/IAssetTranscoder.cs rename to source/CapriKit.AssetPipeline/IAssetTranscoder.cs index 29859f1..8e885ec 100644 --- a/source/CapriKit.AssetPipeline/vNext/IAssetTranscoder.cs +++ b/source/CapriKit.AssetPipeline/IAssetTranscoder.cs @@ -1,7 +1,7 @@ using CapriKit.IO; using System.Buffers; -namespace CapriKit.AssetPipeline.vNext; +namespace CapriKit.AssetPipeline; /// /// Interface for classes that builds assets (such as texture, models and sound effects) and load them diff --git a/source/CapriKit.AssetPipeline/vNext/NoSettingsTranscoder.cs b/source/CapriKit.AssetPipeline/NoSettingsTranscoder.cs similarity index 94% rename from source/CapriKit.AssetPipeline/vNext/NoSettingsTranscoder.cs rename to source/CapriKit.AssetPipeline/NoSettingsTranscoder.cs index 0810a2a..a86ec8c 100644 --- a/source/CapriKit.AssetPipeline/vNext/NoSettingsTranscoder.cs +++ b/source/CapriKit.AssetPipeline/NoSettingsTranscoder.cs @@ -1,7 +1,7 @@ using CapriKit.IO; using System.Buffers; -namespace CapriKit.AssetPipeline.vNext; +namespace CapriKit.AssetPipeline; public readonly struct NoSettings; From c9c853b46ca2eb436a34df7b89afebadf0414115 Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Fri, 14 Aug 2026 14:17:44 +0200 Subject: [PATCH 38/53] Add first E2E asset loading test (without hot reloading) --- .../Shaders/VertexShaderTranscoder.cs | 1 - source/CapriKit.AssetPipeline/AssetManager.cs | 6 +- .../AssetPipeline/AssetManagerTests.cs | 80 ++++++++++++++++++- 3 files changed, 77 insertions(+), 10 deletions(-) diff --git a/source/CapriKit.AssetPipeline.DirectX11/Shaders/VertexShaderTranscoder.cs b/source/CapriKit.AssetPipeline.DirectX11/Shaders/VertexShaderTranscoder.cs index 658699d..f1824d0 100644 --- a/source/CapriKit.AssetPipeline.DirectX11/Shaders/VertexShaderTranscoder.cs +++ b/source/CapriKit.AssetPipeline.DirectX11/Shaders/VertexShaderTranscoder.cs @@ -1,4 +1,3 @@ -using CapriKit.AssetPipeline.vNext; using CapriKit.DirectX11; using CapriKit.DirectX11.Resources.Shaders; using CapriKit.IO; diff --git a/source/CapriKit.AssetPipeline/AssetManager.cs b/source/CapriKit.AssetPipeline/AssetManager.cs index 436dac4..6c817fa 100644 --- a/source/CapriKit.AssetPipeline/AssetManager.cs +++ b/source/CapriKit.AssetPipeline/AssetManager.cs @@ -9,10 +9,6 @@ namespace CapriKit.AssetPipeline; public sealed partial class AssetManager : IDisposable { - // TODO: ensure that the transcoders and hot-reload manager are thread safe - // TODO: what if an asset is requested multiple times? Do we keep track - // of in-flight loading? Same for hot-reloading. - private readonly ILogger Logger; private readonly ScopedFileSystem FileSystem; private readonly AssetCache Cache; @@ -58,7 +54,7 @@ public void RegisterTranscoder(IAssetTranscoder - public AssetHandle Load(AssetId id, TSettings settings) + internal AssetHandle Load(AssetId id, TSettings settings) where TAsset : class { var handle = new AssetHandle(); diff --git a/source/CapriKit.Tests/AssetPipeline/AssetManagerTests.cs b/source/CapriKit.Tests/AssetPipeline/AssetManagerTests.cs index 00f0044..316fe7f 100644 --- a/source/CapriKit.Tests/AssetPipeline/AssetManagerTests.cs +++ b/source/CapriKit.Tests/AssetPipeline/AssetManagerTests.cs @@ -1,10 +1,82 @@ -using System; -using System.Collections.Generic; -using System.Text; +using CapriKit.AssetPipeline; +using CapriKit.IO; +using CapriKit.IO.Streams; +using CapriKit.Tests.TestUtilities; +using Microsoft.Extensions.Logging.Abstractions; +using System.Buffers; namespace CapriKit.Tests.AssetPipeline; internal class AssetManagerTests { - + private DirectoryPath? WorkingDirectory; + private readonly FilePath AssetFile = new("Hello.txt"); + private const string TranscoderText = "Hello World"; + + [Before(Test)] + public void Setup() + { + WorkingDirectory = FileSystemUtilities.CreateTemporaryDirectory(); + var fileSystem = new FileSystem().ScopedTo(WorkingDirectory); + using var stream = fileSystem.CreateReadWrite(AssetFile); + using var writer = new StreamWriter(stream); + writer.Write(TranscoderText); + } + + [After(Test)] + public void TearDown() + { + Directory.Delete(WorkingDirectory, true); + } + + [Test] + public async Task LoadAsset() + { + await Assert.That(WorkingDirectory).IsNotNull(); + + var logger = NullLoggerFactory.Instance; + var fileSystem = new FileSystem().ScopedTo(WorkingDirectory); + var assetManager = new AssetManager(NullLoggerFactory.Instance, fileSystem); + + var transcoder = new TestTranscoder(); + assetManager.RegisterTranscoder(transcoder); + + var id = new AssetId(string.Empty, AssetFile); + + var builder = new AssetBundleBuilder(assetManager); + var handle = builder.Load(id, default); + var loader = builder.Build(resolver => new TestBundle(resolver.Get(handle))); + + TestBundle? bundle = null; + await Assert.That(() => + { + assetManager.Update(); + return loader.IsReady(out bundle); + }) + .Eventually(v => v.IsTrue(), TimeSpan.FromSeconds(5)); + + await Assert.That(bundle).IsNotNull(); + await Assert.That(bundle.Text).IsEqualTo(TranscoderText); + } + + private record TestBundle(string Text); + + private class TestTranscoder() : NoSettingsTranscoder(Guid.Parse("{AC2D4E77-0D98-43B2-B1D2-35B0E9F5742B}"), 1) + { + public override string Decode(AssetId id, ref SequenceReader reader) + { + return reader.ReadString(); + } + + public override async Task Encode(AssetId id, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer) + { + var text = await fileSystem.ReadAllText(id.Path); + writer.Write(text); + } + + public override void HotSwap(string instance, string newParts) + { + throw new NotImplementedException(); + } + } } From 072cf586a0b10b523d272ae34bfad4719552d93f Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Fri, 14 Aug 2026 14:55:06 +0200 Subject: [PATCH 39/53] Fix IO issue, work on report for newest version --- Research/AssetPipelineVNextReview.html | 675 ++++++++++++++++++ .../AssetManagerExtensions.cs | 14 - source/CapriKit.AssetPipeline/AssetManager.cs | 19 +- source/CapriKit.IO/FileSystem.cs | 6 +- source/CapriKit.IO/InMemoryFileSystem.cs | 6 +- 5 files changed, 695 insertions(+), 25 deletions(-) create mode 100644 Research/AssetPipelineVNextReview.html delete mode 100644 source/CapriKit.AssetPipeline.DirectX11/AssetManagerExtensions.cs diff --git a/Research/AssetPipelineVNextReview.html b/Research/AssetPipelineVNextReview.html new file mode 100644 index 0000000..5193e52 --- /dev/null +++ b/Research/AssetPipelineVNextReview.html @@ -0,0 +1,675 @@ + + + + + +Asset Pipeline vNext Review + + + + +
+
+
CapriKit · feature/asset_pipeline · c9c853b
+

Asset Pipeline vNext Review

+

Four independent reviews — correctness, thread-safety, API ergonomics, implementation size — of + source/CapriKit.AssetPipeline and its DirectX11 consumer. Findings below are merged by defect, so one entry may + carry badges from several reviews that hit it independently.

+
+ Reviewed 1,195 lines / 13 files + 3 DX11 files + Entry point AssetManagerTests.LoadAsset + Date 2026-08-14 + Repros executed 6 of 8 critical/high claims +
+ +
+
Solution build
FAIL
DX11 project, CS1061
+
Critical
4
3 with executed repros
+
High
6
5 in hot reload
+
Medium / Low
14
triage after the above
+
Hot reload
0%
never ran end‑to‑end
+
Removable
47%
1,195 → ~630 lines
+
+
+
+ +
+
+

The headline

+

The passing test and the three critical bugs are not in tension — the test only ever exercises the + first-run path, and every critical defect lives on a path it cannot reach.

+ +

[Before(Test)] creates a fresh temporary directory for each test, so no .cka build artifact ever + exists when the test runs. That means TryDecodeBuildMetaData always returns null, the up-to-date + branch is never taken, and the missing return that corrupts it is invisible. The test also loads exactly + one asset, of type string — which is not IDisposable, so the disposal bug cannot + manifest — and it never calls Dispose on the manager, which would throw. One green test, four criticals, + no contradiction.

+ +
+ What this means for the design +

None of the four criticals is a flaw in the vNext design. The promise/bundle/pull model from + AssetPipelineLoadingGroupsV2.md holds up; two of the criticals are the design's own written requirements + (“N−1 extra leases”, “faulted counts as resolved”) simply not implemented yet, and one is a + missing return statement. The architecture is sound. The wiring is not finished.

+
+
+
+ +
+
+

Blockers

+

Fix these four before anything else. Nothing downstream can be trusted while they stand.

+ +
+
B1The solution does not build — the only real consumer calls a two-iteration-old API
+
CapriKit.AssetPipeline.DirectX11/AssetManagerExtensions.cs:12
+
verifiedcorrectnessthreadingapisize
+ 
error CS1061: 'AssetManager' does not contain a definition for 'Load'
+               and no accessible extension method 'Load' accepting a first
+               argument of type 'AssetManager' could be found
+

LoadVertexShader still calls Load(id, transcoder, default) returning Task<IVertexShader>. + The current Load is internal and takes (AssetId, TSettings). Because only + CapriKit.Tests has InternalsVisibleTo, the entire D3D11 side of the pipeline is unreachable from + outside the assembly — which is precisely why none of the races below have bitten a real renderer yet.

+

dotnet build CapriKit.AssetPipeline.csproj succeeds on its own, so this is invisible unless you build the + solution. Downstream convenience extensions can no longer hang off AssetManager at all; the public entry + point is now AssetBundleBuilder.

+ Fix + 
public static AssetHandle<IVertexShader> LoadVertexShader(this AssetBundleBuilder builder, AssetId id)
+    => builder.Load<IVertexShader, NoSettings>(id, default);
+

The per-call new VertexShaderTranscoder(device) must also move to a one-time registration — + RegisterTranscoder throws on the second call for the same asset type.

+
+ +
+
B2Missing return: every asset is rebuilt on every boot, then crashes Update()
+
AssetManager.cs:106-122 — no return after the up-to-date branch at :111
+
repro executedcorrectnessthreadingsize
+

The if (build != default && IsUpToDate(...)) block writes to Incoming and then + falls through into the rebuild path, so a current asset is decoded, then re-encoded and decoded again, + and published to the channel twice.

+ Failure scenario — executed +

Build Hello.txt once so Hello.txt.cka exists. Restart with a fresh AssetManager over + the same directory. Update() drains message 1, resolves the handles, does + Outstanding.Remove(id). It then drains message 2 and hits Outstanding[id]:

+ 
System.Collections.Generic.KeyNotFoundException: The given key
+'AssetId { Key = , Path = Hello.txt }' was not present in the dictionary.
+   at CapriKit.AssetPipeline.AssetManager.Update() in AssetManager.cs:line 145
+

Because the staleness fast path has therefore never once executed, the whole envelope and staleness subsystem is + far less load-bearing than its line count suggests — which lowers the risk of restructuring it (see S2).

+ Fix — both halves + 
    LogLoadedFromFile(Logger, id);
+    return;                                  // <-- missing
+
+// and harden the drain so a stray message can never kill the frame loop:
+if (!Outstanding.Remove(id, out var handles)) { continue; }
+
+ +
+
B3Two handles for one asset id hand out an already-disposed instance
+
AssetManager.cs:145-151 (materializer called per handle) · AssetCache.cs:36-43
+
repro executedcorrectnessthreading
+ 
var handles = Outstanding[id];
+foreach (var handle in handles)
+{
+    var asset = materializer();   // <-- called once PER HANDLE
+    handle.Resolve(asset);
+}
+

The materializer closes over a single decoded instance. On the second call PutOrLease finds the entry already + present and enqueues new Entry(id, candidate, 0) for disposal — but candidate + is entry.Asset, the live object. Cache.Collect(), three lines later in the same frame, + disposes it. Both handles now hold a disposed asset that is still in Entries with refcount 2.

+ Failure scenario — executed +

Two handles for one id: same instance: True, A.IsDisposed: True, B.IsDisposed: True. Under load — + 8 threads × 100 loads over 8 ids with a pumping Update() — + 585 of 800 bundles received a disposed asset, with zero exceptions raised. Nothing signals the corruption. + For a VertexShaderTranscoder this is a released ID3D11VertexShader handed to gameplay code.

+

The author's own ExampleAssets sample (AssetBundleLoader.cs:71-72) requests the same + AssetId twice, so the shipped usage sketch triggers this directly.

+ Fix — this is the “N−1 extra leases” note from the V2 design + 
if (!Outstanding.Remove(id, out var handles)) { continue; }
+var asset = materializer();                  // exactly once
+Cache.AddLeases(id, handles.Count - 1);      // new: refcount += n under Lock
+foreach (var handle in handles) { handle.Resolve(asset); }
+

Additionally guard PutOrLease with ReferenceEquals(entry.Asset, candidate) and skip the enqueue. + Queueing a live object for disposal should be impossible, not merely unreachable.

+
+ +
+
B4One failed load permanently kills Update() for the rest of the process
+
AssetManager.cs:83-88 · LightweightChannel.cs:48 · AssetManager.cs:140-153
+
repro executedcorrectnessthreading
+

LightweightChannel.Error is set once and never cleared; TryRead rethrows it + whenever the queue is empty. The worker's failure path writes there, so Update() throws on every subsequent + frame — from inside lock (RequestLock), before Cache.Collect() and + HotReloadManager.Update() are ever reached.

+ Failure scenario — executed +

Request one missing file, then pump: Update() threw 199 of 200 times. Pending disposals + accumulate forever, no other asset can ever complete, hot reload stops, and the failed id stays in + Outstanding so its handles never resolve — IsReady returns false forever + rather than reporting the error.

+ Fix +

Reserve the channel's sticky error slot for “the pump itself is broken”, and give the handle the faulted state + the V2 design already specified (“faulted counts as resolved”):

+ 
internal ExceptionDispatchInfo? Error;
+internal bool IsResolved => Volatile.Read(ref value) is not null || Error is not null;
+

Route the failure per-id so it surfaces from its bundle's IsReady/Wait, once, instead of + poisoning the shared pump.

+
+
+
+ +
+
+

Hot reload has never worked

+

Five separate defects compound here. The subsystem is the largest in the pipeline (287 lines, 24%) and the + least tested — its only test constructs the class and asserts nothing.

+ +
+
H1Every reload throws: the encoder disposes a stream it does not own
+
AssetEncoder.cs:24 · symptom at HotReloadable.cs:39-43 · same bug in AssetDecoder.cs:24
+
repro executedcorrectness
+

using var output = outputStreamOverride ?? fileSystem.CreateReadWrite(outputPath) takes ownership of a caller's + stream. PipeWriter.Create(stream) defaults to LeaveOpen = false, so + CompleteAsync() already disposed it; the using disposes it again.

+ 
REPRO4 caught: ObjectDisposedException: Cannot access a closed Stream.
+   at System.IO.MemoryStream.Seek(Int64, SeekOrigin)
+

So HotReloadable.Reload throws at stream.Seek(0, SeekOrigin.Begin) on every hot + reload. Combined with H2 below, hot reload has never completed once.

+ Fix + 
var writer = PipeWriter.Create(output, new StreamPipeWriterOptions(leaveOpen: true));
+try { /* snip: write */ }
+finally { if (outputStreamOverride is null) { output.Dispose(); } }
+
+ +
+
H2isReloading latches true forever, and is read across threads without a barrier
+
HotReloadManager.cs:32, 98, 146, 175-185
+
correctnessthreading
+

isReloading = true is set unconditionally at :146, but is only cleared inside the + FireAndForget callbacks attached to target?.Reload(...). When target is + null — the case the code's own comment calls normal — the null-conditional + short-circuits and nothing is ever attached. Update() then returns early on every subsequent frame: + no file-change draining, no reloads, and HotSwapPending() never runs again either.

+

Separately, the flag is written from a thread-pool continuation and read on the main thread as a plain + bool — a data race the JIT may resolve by keeping the read in a register.

+ Fix + 
private volatile bool isReloading;
+// ...
+if (target is null) { isReloading = false; return; }   // before dispatching
+
+ +
+
H3Hot reload can HotSwap an asset that Collect already disposed
+
HotReloadable.cs:19,31,35 · AssetCache.cs:88-92
+
threading
+

WeakReference.IsAlive answers “the GC has not collected it”, not “it has not been + disposed”. Return moves an entry to PendingDispose (still strongly reachable); + Collect disposes it; the object then stays GC-alive for an arbitrary time. ReloadOne sees + IsAlive == true, picks it as target, and HotSwap runs against a released COM object.

+ Fix +

Liveness must mean “the cache still holds a lease”. Have the cache call + HotReloadManager.Untrack(id) on eviction, so IsAlive never depends on GC timing.

+
+ +
+
H4Tracking dictionaries are read outside the lock that guards their writes
+
HotReloadManager.cs:155-171 (Tracked) · :122-130 (Dependents)
+
threading
+ 
lock (TrackingLock) { candidates = Tracked[id]; }   // copies the REFERENCE only
+// ...
+foreach (var candidate in candidates) { ... }        // iterated unlocked
+

The lock covers the dictionary lookup and nothing else, while Track mutates that very list under the lock. + DrainFileChanges reads Dependents with no lock at all — and its XML doc + (“can only be used by one thread at a time”) directly contradicts Track's doc + (“thread-safe, multiple threads can call this method”).

+

Latent, not live. Track is currently only reachable from Update(), so + everything runs on the main thread today. It breaks the moment materialization moves off it — and the documentation + already promises the guarantee the code does not provide. Resolve the contradiction in one direction and delete + the // TODO: ensure that HotReloadManager.Track is thread safe comments either way.

+
+ +
+
H5The dedupe guard compares two different types, so it is always false
+
HotReloadManager.cs:68
+
correctnessthreading
+ 
if (assets.Any(a => object.ReferenceEquals(a, asset))) { return; }
+//                  ^ HotReloadable        ^ Asset<TAsset,TSettings>
+

Different types by construction, never reference-equal. It compiles, and the documented guarantee + “registering the same asset multiple times is safe” does not hold. Compounding it, + ReloadOne builds a pruned list at :162-170 and then throws it + away — so Tracked grows monotonically for the process lifetime, and dead ids keep feeding + PendingRebuilds, which is exactly the input that triggers H2's permanent latch.

+
+
+
+ +
+
+

Remaining findings

+
+ + + + + + + + + + + + + + + + + + +
IDSeverityFindingLocation
M1HighAssetManager.Dispose always throws (nothing ever calls Cache.Return), so HotReloadManager.Dispose never runs and the OS FileSystemWatcher leaks on every shutdownAssetManager.cs:159
M2MediumIsReady re-materialises the bundle on every call for struct bundles — result == null on a notnull type parameter boxes and is always false. Reintroduces the exact defect V2 was written to fixAssetBundleLoader.cs:45
M3MediumAssetHandle.value/isResolved are published without a fence the reader participates in. On ARM64 a reader can see isResolved == true with a stale null valueAssetHandle.cs:7-19
M4MediumReloadOne starts the rebuild on the calling thread — unlike Load, it is not wrapped in Task.Run, so a synchronous D3DCompile can run on the main threadHotReloadManager.cs:175
M5MediumDisposal is uncoordinated with in-flight work: HotReloadManager.Dispose clears its dictionaries without the lock; AssetManager.Dispose neither drains the channel nor awaits outstanding requestsAssetManager.cs:159, HotReloadManager.cs:211
M6MediumUnload bypasses RequestLock: unloading an in-flight asset throws, then the load completes anyway and takes a lease nobody returns. This is V2's open cancellation question, still unansweredAssetManager.cs:126
M7MediumRegistering a transcoder after a load of that type throws on the worker — which via B4 kills the manager permanently. “Thread-safe” here means “won't corrupt the dictionary”, not “order-independent”AssetManager.cs:39,222
M8MediumHot reload discards hot.BuildMetaData and never re-tracks, so an .hlsl that gains a new #include never registers that file until process restartHotReloadable.cs:33-46
L1LowSuccessful hot reload is logged at LogLevel.Error (copy-paste from the adjacent failure message)HotReloadManager.cs:226
L2LowA failed encode leaves a truncated .cka. Self-heals via the blanket catch, but write-to-temp-then-move would make “a .cka on disk is complete” an invariantAssetEncoder.cs:24-35
L3LowDebug.Assert is the only double-resolve guard, so in Release a duplicate publish silently overwrites a resolved handleAssetHandle.cs:16
L4Low*.cka artifacts are written beside the source files and are not in .gitignore, so a first run fills the art folder and dirties git statusAssetUtilities.cs:7, .gitignore
L5LowNo ConfigureAwait(false) anywhere, though CapriKit.IO uses it on all of its awaits. Harmless only while no SynchronizationContext exists — and not harmless for M412 awaits, 4 files
L6Lowpublic sealed record ExampleAssets ships in the NuGet package; AssetHandle's fields are internal protected with a public constructor, so external code can forge a resolved handleAssetBundleLoader.cs:66, AssetHandle.cs:7
+
+ +
+ Checked and found clean — recorded so these are not re-reviewed later +
    +
  • Envelope round trip. Field order in AssetEncoder matches both decode paths exactly; + SkipPayload advances by precisely payloadLength; endianness is symmetric; + SliceUnread correctly advances the outer reader past a sub-reader that under-reads; + ArrayPool.Rent over-allocates but SequenceReaders.Create(buffer, 0, length) bounds it.
    • +
  • Lock ordering. The only nestings are RequestLock → Cache.Lock and + RequestLock → TrackingLock, consistently. No deadlock exists today — but the invariant is written + down nowhere, and is one careless Unload inside a cache callback away from breaking.
    • +
  • AssetCache.Collect's TryEnter fast path. Correct. The + try/finally releases the lock on the early return, PendingDispose is monotone so nothing is + lost, and deferring a disposal by one frame is the intent. Worth a comment so nobody “fixes” it into a + blocking lock.
    • +
  • Disposing outside the lock. DisposeDrainedItems running outside Lock is + deliberate and correct — it is what makes an asset whose Dispose calls back into Unload + safe.
    • +
  • Path scoping. Watcher events are re-relativised to BasePath, matching the relative paths + VirtualFileSystemSpy records, so Dependents lookups hit. The .cka is written + through the raw file system, not the spy, so an asset never lists its own build output as a dependency — no + infinite rebuild loop.
    • +
  • Task.Run / FireAndForget shape. Binds the unwrapping overload, so + FireAndForget awaits the real work rather than a proxy. Correct.
    • +
+
+
+
+ +
+
+

API: the boilerplate problem

+

Loading one asset costs four mentions of the same thing — the AssetId, the + Load line, the resolver.Get argument, and the bundle record member. Two of the four are removable + today. The third and fourth are inherent to “declare N loads, then build one object”, and only a source generator + collapses them.

+ +
+
+

Today

+ 
var id      = new AssetId(string.Empty, AssetFile);
+var builder = new AssetBundleBuilder(assetManager);
+var handle  = builder.Load<string, NoSettings>(id, default);
+var loader  = builder.Build(r => new TestBundle(r.Get(handle)));
+
+TestBundle? bundle = null;
+// poll assetManager.Update() until loader.IsReady(out bundle)
+
+
+

Proposed (P1–P5, no codegen)

+ 
var bundle = assetManager.CreateBundle();
+var text   = bundle.Load<string>(AssetFile);
+var loader = bundle.Build(text, t => new TestBundle(t));
+
+using var assets = loader.Wait();   // bootstrap: pumps Update() itself
+
+
+ +
+ The honest limit +

For a realistic renderer — 3 shaders + 4 textures — these changes save about 22 characters per line + but the structure does not shrink. Worse, the factory is positional across same-typed assets, so the three + IVertexShader handles are mutually assignable: 144 wrong orderings compile and render silently + wrong. That is the real argument for the source generator (P8) — not brevity.

+
+ +
+ + + + + + + + + + + + +
IDChangeWinCostBreaking
P1Load<TAsset>(id) — erase TSettings at registrationBiggest per-character win; NoSettings leaves user code entirely~20 lnyes
P2AssetId: path first, key optional, implicit from FilePath onlyKills string.Empty and the silent argument-swap hazard~15 lnyes, 3 sites
P3assetManager.CreateBundle() instead of new AssetBundleBuilder(mgr)Discoverable from the object you already hold; V2 specified it3 lntrivially
P4AssetManager(ILoggerFactory, DirectoryPath) overloadCallers stop writing new FileSystem().ScopedTo(dir)4 lnno
P5Build overloads for arity 1–3 taking handles directlyResolver disappears; redemption becomes structurally impossible to get wrong~12 lnadditive
P6Wait() (block/pump) + IDisposable on the bundleCloses the two holes the code itself flags; makes the bundle the lifetime unit~40 lnyes
P7Per-bundle faulted state instead of the sticky channel errorFixes B4; a bad file stops poisoning every other load~15 lnno
P8[AssetBundle] source generatorOnly thing that collapses 4 mentions to 1 and kills the ordering hazard~300 lnadditive
+
+ +

Stop the handle-passing overloads at arity 3. At arity 7, + Build(a,b,c,d,e,f,g, (a,b,c,d,e,f,g) => new X(...)) mentions each asset three times inside + Build versus once for r.Get(a). Keep both paths; document the crossover. + AssetHandleResolver does earn its keep — it is the capability token that makes redeeming a handle outside + materialization unrepresentable.

+ +
+ Considered and rejected — brevity that costs type safety +
    +
  • AssetHandle<T>.Value / IsLoaded. Removes one mention per asset. + Reject — this is the exact invariant both design docs exist to protect. A readable + .Value re-admits “not loaded yet” into every consumer's types and puts the per-frame check back + into every system.
    • +
  • Nullable bundle members with placeholder fallback. Reject for shaders — a + shader's interface is its identity, so no valid placeholder exists.
    • +
  • Ordered redemption (r.Next<T>()). Reject — silently couples + the factory to Load call order, strictly worse than explicit handle identity.
    • +
  • Reflection matching handles to record parameters. Reject — moves every arity and + type error to runtime, defeating the “compiler forces the constructor to match” payoff that the design docs + call the whole point of hardcoding requirements.
    • +
+
+
+
+ +
+
+

Implementation size

+

1,195 lines across 13 files. A recommended set of seven changes projects to ~630 lines in 9 + files — a 47% reduction — with hot reload no longer in the shipping path.

+ +
+ + + + + + + + + + + + +
SubsystemFilesLinesShareVerdict
Hot reload228724%Largest, least tested, 5 defects — and dev-only by design
Orchestration (AssetManager)124921%Keep; trim with S5/S7
Envelope encode/decode323420%Reads every file twice; merge
Cache / refcounting118115%Deferred disposal may not be load-bearing
Bundles / handles211610%Keep; needs faulted state added back
Transcoder abstraction2928%The TSettings arity is the expensive half
Model + glue2363%—
Total131,195100%Projected after: ~630
+
+ +
+ + + + + + + + + + + + +
IDSimplificationLinesRiskCapability lost
S1Delete dead weight: ExampleAssets, ServiceCollectionExtensions, Unload, the discarded pruned list, the leak-throw in Cache.Dispose−51nonenone
S2One AssetEnvelope over byte[], not streams. Move dependencies before the payload so staleness reads only the prefix; merge the two decode methods that read the same file twice−141lownone — staleness gets faster
S3Shrink AssetCache: drop PendingDispose, Collect, and the TryEnter path; dispose on zero, outside the lock−106low–medworker-thread deferred dispose
S4Handle/bundle trim — delete the empty AssetBundleLoader base, collapse two fields to one object?; add ~8 lines back for the faulted state−15/+8nonenone
S5Drop the transcoder registry; pass the transcoder to Load. Keying on typeof(TAsset) silently forbids two transcoders for one type — negative convenience−30lowload-by-type-alone
S6Hot reload via closures instead of a class hierarchy, and opt-in behind a flag so shipping builds never construct it−155med–highreload throttling, weak pruning
S7Delete the TSettings type parameter; fold settings into the transcoder's Version−83mediumper-request settings
Tiered: S1+S2+S4 → 988 · +S3+S5 → 852 · +S6+S7 → ~630−573
+
+ +

On S7, note the per-request settings capability is already largely illusory: Transcoders is keyed by + asset type so one type has exactly one transcoder, and ToEncodedFilePath does not include settings — so + loading one id with two different settings already ping-pongs rebuilds against the same .cka.

+ +

Keep as is — load-bearing despite looking complex

+
    +
  • LightweightChannel + main-thread Update() drain. The threading model itself, and + the only place a transcoder's ID3D11DeviceContext step can legally run.
    • +
  • Outstanding: Dictionary<AssetId, List<AssetHandle>>. This is V2's fix for the + 1:N in-flight relationship. Collapsing it to one handle per id reintroduces the hang V2 was written to fix.
    • +
  • VirtualFileSystemSpy dependency capture. Four lines in the encoder buy the entire + #include staleness and hot-reload trigger story. This is the reuse model the rest of the pipeline should + imitate.
    • +
  • PutOrLease returning the winner and disposing the loser. Looks like paranoia; is not. + Load only dedupes inside the lock, and hot reload decodes concurrently with normal loads.
    • +
  • AssetHandle<T> as an empty generic subclass. A zero-cost compile-time type tag — + what makes “not yet loaded” unrepresentable. The runtime-Type alternative is longer and weaker.
    • +
  • HotSwap on the transcoder. Why no AssetRef<T> indirection layer is needed + at all — game code holds the object directly.
    • +
+
+
+ +
+
+

Suggested order

+

A genuine sequence — each step makes the next one safe or cheaper.

+
    +
  1. B1 — make the solution build. +

    Nothing else can be verified end-to-end while the only real consumer is uncompilable. Decide here whether + AssetManagerExtensions is deleted or reshaped around the builder (P5 answers this).

    2. +
  2. B2, B3, B4 — the three criticals. +

    Two return-shaped fixes and one lease-counting fix. All three have executed repros, so each can be turned + into a regression test immediately.

    3. +
  3. Write the tests those repros became. +

    Second-run/up-to-date load, two handles for one id, and a failing load followed by ten Update()s. These are + the three paths the current test cannot reach, and they are what turn the rest of this list from review comments into a + safety net.

    4. +
  4. P1–P5 — the API shape, while call sites are still nearly zero. +

    At 0.1.0-alpha these breaking changes are free today and expensive later. P2 in particular closes a + silent-swap hazard that no amount of documentation fixes.

    5. +
  5. S1, S2, S4 — the zero-capability-loss deletions. +

    Takes the pipeline to ~988 lines and removes the double file read on every load. S2 is low-risk precisely because + B2 proves the staleness path has never run.

    6. +
  6. Decide hot reload's fate (H1–H5, S6). +

    It has never worked, so there is no regression to fear and full freedom to redesign. Make it opt-in first — five + lines — so it leaves the shipping mental model, then fix or rewrite behind that flag.

    7. +
  7. Settle the threading contract (H4, M3, M5). +

    The table below is the current state, not a specification. Pick one direction, write it in the XML docs, and delete the + // TODO comments that promise the other.

    8. +
+
+
+ +
+
+

The threading model, as implemented

+

Not as documented — as it actually behaves today. Worth keeping as the module's reference page, since the + XML docs and the code currently disagree in three places.

+
+ + + + + + + + + + + + + + + +
ClassFieldWritersReadersGuardVerdict
AssetManagerTranscodersanypoolconcurrent typesafe
Outstandingany + mainsameRequestLocksafe, but indexed by a worker-supplied key
Incomingpoolmainconcurrent + volatileerror slot sticky (B4)
AssetCacheEntries, RefCountanyanyLock alwayssafe
PendingDisposeanymainLock / TryEntercan contain a live object (B3)
AssetHandlevalue, isResolvedany, under lockmain, unlockednone on readunsynchronised publication (M3)
Ownerbuilder threadmainnonesame class of issue
HotReloadManagerTrackedmain in practicemainlookup onlyinner list iterated unlocked (H4)
Dependentsunder lockno lockinconsistentdoc contradicts code (H4)
isReloadingmain sets, pool clearsmainnonerace + latches true (H2)
PendingReloadspoolmainconcurrent typesafe
+
+

In one line: Load, Unload and RegisterTranscoder are genuinely callable from any + thread; RequestAsset, Encode and Decode run on the pool; and — despite the + documentation — MaterializeAsset, PutOrLease, Track, IsReady and + Get all run on the main thread. HotReloadable.Reload's synchronous prefix runs on the main thread + too, which is not intended (M4).

+
+
+ +
+
+

Method. Four independent reviews ran in parallel against commit c9c853b, each reading the full + pipeline source plus its CapriKit.IO and CapriKit.Concurrency dependencies. Findings were merged by + defect and cross-checked against a manual read; the build failure and the two behaviour-changing claims about it were verified + directly. Six of the eight critical and high findings were confirmed by executing a repro; the rest are marked as reasoned from + the code. All temporary test files were removed and the working tree left clean.

+

Prior art. Supersedes AssetPipelineReview.md and + AssetPipelineRewriteReview.md, which reviewed earlier iterations. The design intent reviewed against is + AssetPipelineLoadingGroupsV2.md, whose model this code implements and whose open questions — cancellation, + hot-reload re-entry into a live bundle — remain open.

+
+
+ + + diff --git a/source/CapriKit.AssetPipeline.DirectX11/AssetManagerExtensions.cs b/source/CapriKit.AssetPipeline.DirectX11/AssetManagerExtensions.cs deleted file mode 100644 index 42dbe28..0000000 --- a/source/CapriKit.AssetPipeline.DirectX11/AssetManagerExtensions.cs +++ /dev/null @@ -1,14 +0,0 @@ -using CapriKit.AssetPipeline.DirectX11.Shaders; -using CapriKit.DirectX11; -using CapriKit.DirectX11.Resources.Shaders; - -namespace CapriKit.AssetPipeline.DirectX11; - -public static class AssetManagerExtensions -{ - public static Task LoadVertexShader(this AssetManager assetManager, Device device, AssetId id) - { - var transcoder = new VertexShaderTranscoder(device); - return assetManager.Load(id, transcoder, default); - } -} diff --git a/source/CapriKit.AssetPipeline/AssetManager.cs b/source/CapriKit.AssetPipeline/AssetManager.cs index 6c817fa..97641ce 100644 --- a/source/CapriKit.AssetPipeline/AssetManager.cs +++ b/source/CapriKit.AssetPipeline/AssetManager.cs @@ -109,17 +109,18 @@ private async Task RequestAsset(AssetId id, TSettings setting Incoming.Write((id, () => MaterializeAsset(upToDateAsset, transcoder))); LogLoadedFromFile(Logger, id); } - - // If not, try to rebuild and load the asset - if (!FileSystem.Exists(id.Path)) + else // If not, try to rebuild and load the asset { - throw new FileNotFoundException("Could not find primary file to build asset from", id.Path); - } + if (!FileSystem.Exists(id.Path)) + { + throw new FileNotFoundException("Could not find primary file to build asset from", id.Path); + } - await AssetEncoder.Encode(id, transcoder, settings, FileSystem); - var freshAsset = await AssetDecoder.Decode(id, transcoder, FileSystem); - Incoming.Write((id, () => MaterializeAsset(freshAsset, transcoder))); - LogBuildAndLoaded(Logger, id); + await AssetEncoder.Encode(id, transcoder, settings, FileSystem); + var freshAsset = await AssetDecoder.Decode(id, transcoder, FileSystem); + Incoming.Write((id, () => MaterializeAsset(freshAsset, transcoder))); + LogBuildAndLoaded(Logger, id); + } } // Thread safe diff --git a/source/CapriKit.IO/FileSystem.cs b/source/CapriKit.IO/FileSystem.cs index 2a34865..2f7a189 100644 --- a/source/CapriKit.IO/FileSystem.cs +++ b/source/CapriKit.IO/FileSystem.cs @@ -94,7 +94,11 @@ public IVirtualFileSystemWatcher Watch(DirectoryPath directory, bool includeSubD public FilePath GetAbsolutePath(FilePath file) => file.ToAbsolute(Environment.CurrentDirectory); - public DirectoryPath GetAbsolutePath(DirectoryPath directory) => directory.ToAbsolute(Environment.CurrentDirectory); + public DirectoryPath GetAbsolutePath(DirectoryPath directory) + { + if (directory.IsAbsolute) { return directory; } + return directory.ToAbsolute(Environment.CurrentDirectory); + } private FileInfo FindOrThrow(FilePath file) { diff --git a/source/CapriKit.IO/InMemoryFileSystem.cs b/source/CapriKit.IO/InMemoryFileSystem.cs index 534e0fb..b3db41d 100644 --- a/source/CapriKit.IO/InMemoryFileSystem.cs +++ b/source/CapriKit.IO/InMemoryFileSystem.cs @@ -97,7 +97,11 @@ public IVirtualFileSystemWatcher Watch(DirectoryPath directory, bool includeSubD public FilePath GetAbsolutePath(FilePath file) => file.ToAbsolute(Environment.CurrentDirectory); - public DirectoryPath GetAbsolutePath(DirectoryPath directory) => directory.ToAbsolute(Environment.CurrentDirectory); + public DirectoryPath GetAbsolutePath(DirectoryPath directory) + { + if (directory.IsAbsolute) { return directory; } + return directory.ToAbsolute(Environment.CurrentDirectory); + } private InMemoryFile FindOrThrow(FilePath file) { From c7517e79c3f2fd5bf04725741809ba647c4a18fd Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Sat, 15 Aug 2026 11:23:22 +0200 Subject: [PATCH 40/53] Work on most findings --- .gitignore | 1 + ...nseToFindingsOfAssetPipelineVNextReview.md | 18 +++ source/CapriKit.AssetPipeline/Asset.cs | 4 +- .../{AssetBundleLoader.cs => AssetBundle.cs} | 55 ++++---- source/CapriKit.AssetPipeline/AssetCache.cs | 7 + source/CapriKit.AssetPipeline/AssetDecoder.cs | 1 - source/CapriKit.AssetPipeline/AssetEncoder.cs | 28 ++-- source/CapriKit.AssetPipeline/AssetHandle.cs | 15 ++- source/CapriKit.AssetPipeline/AssetManager.cs | 41 +++++- .../HotReloadManager.cs | 124 +++++++++++------- .../CapriKit.AssetPipeline/HotReloadable.cs | 4 +- .../Async/FireAndForgetExtensions.cs | 37 ++---- .../Primitives/LightweightChannel.cs | 22 ++-- .../AssetPipeline/AssetManagerTests.cs | 20 ++- .../Async/FireAndForgetExtensionsTests.cs | 9 +- .../Primitives/LightweightChannelTests.cs | 28 +--- 16 files changed, 248 insertions(+), 166 deletions(-) create mode 100644 Research/ResponseToFindingsOfAssetPipelineVNextReview.md rename source/CapriKit.AssetPipeline/{AssetBundleLoader.cs => AssetBundle.cs} (55%) diff --git a/.gitignore b/.gitignore index c116594..9eb4779 100644 --- a/.gitignore +++ b/.gitignore @@ -2,3 +2,4 @@ .vs .pack .claude/*.local.json +*.cka diff --git a/Research/ResponseToFindingsOfAssetPipelineVNextReview.md b/Research/ResponseToFindingsOfAssetPipelineVNextReview.md new file mode 100644 index 0000000..ac62630 --- /dev/null +++ b/Research/ResponseToFindingsOfAssetPipelineVNextReview.md @@ -0,0 +1,18 @@ +I made changes to fix issues B1-B4, please verify. Note that for B4 I do not mind if one failed load kills the game, but I now added a way to handle that so that the game can at least throw a complete error message. + +I made changes to fix issues H1, H2 and H4, please verify + +For H3 "Hot reload can HotSwap an asset that Collect already disposed": I would like an example of how the HotReloadManager and Cache can work better together I think that will also help me with H5 The dedupe guard compares two different types, so it is always false. How to fix this can I use the cache as some sort of authority here? + +M1: I've changed how assets are disposed, this can now only be done for a bundle at a time. (see AssetManager.Unload). Of course in the tests nobody unloads the assets yet. But is this mechanism sound and thread-safe? + +M2, M3, M4: I think I fixed these, please check + +Keep M5 as a TODO in your next report. I need to look at that later. + +M6, I've changed how unloading works, is this now fixed? + +M7: Ignore that for now, users are supposed to initialize the asset manager with all transcoders on start-up + + +M8: I think I fxed this, but there is now a lot of locking going on in HotReloadManager, can we make this simpler or at least more explicit. Hot reloading is very rare so maybe its better to use the concurrent collection types more often? diff --git a/source/CapriKit.AssetPipeline/Asset.cs b/source/CapriKit.AssetPipeline/Asset.cs index bf4033d..83daf3c 100644 --- a/source/CapriKit.AssetPipeline/Asset.cs +++ b/source/CapriKit.AssetPipeline/Asset.cs @@ -5,9 +5,9 @@ namespace CapriKit.AssetPipeline; /// /// Unique asset identifier /// -/// Optional key to a sub-resources in Path. /// Virtual file path that points to the file the asset originates from. -public record AssetId(string Key, FilePath Path); +/// Optional key to a sub-resources in Path. +public record AssetId(FilePath Path, string Key = ""); internal record Asset(AssetId Id, TAsset Value, AssetBuildMetaData BuildMetaData) where TAsset : class; diff --git a/source/CapriKit.AssetPipeline/AssetBundleLoader.cs b/source/CapriKit.AssetPipeline/AssetBundle.cs similarity index 55% rename from source/CapriKit.AssetPipeline/AssetBundleLoader.cs rename to source/CapriKit.AssetPipeline/AssetBundle.cs index 8b94219..c34b06d 100644 --- a/source/CapriKit.AssetPipeline/AssetBundleLoader.cs +++ b/source/CapriKit.AssetPipeline/AssetBundle.cs @@ -2,9 +2,15 @@ namespace CapriKit.AssetPipeline; -public sealed class AssetBundleBuilder(AssetManager assetManager) +public sealed class AssetBundleBuilder { private readonly List Handles = []; + private readonly AssetManager assetManager; + + internal AssetBundleBuilder(AssetManager assetManager) + { + this.assetManager = assetManager; + } public AssetHandle Load(AssetId id, TSettings settings) where TAsset : class @@ -20,6 +26,7 @@ public AssetBundleLoader Build(Func(factory, Handles); foreach (var handle in Handles) { + bundle.Add(handle.Id); handle.Owner = bundle; } @@ -27,50 +34,48 @@ public AssetBundleLoader Build(Func AssetSet = []; + + internal void Add(AssetId id) => AssetSet.Add(id); + internal bool IsActive { get; set; } = true; + public IReadOnlySet Assets => AssetSet; } public sealed class AssetBundleLoader(Func factory, IReadOnlyList handles) - : AssetBundleLoader + : AssetBundle where TBundle : notnull { + private bool isReady; private TBundle? result; // Single threaded! // TODO: can we reduce the number of things we need to check each frame? public bool IsReady([NotNullWhen(true)] out TBundle? value) { - if (result == null) + if (isReady) { - foreach (var handle in handles) + value = result!; + return true; + } + + foreach (var handle in handles) + { + if (!handle.IsResolved) { - if (!handle.IsResolved) - { - value = default; - return false; - } + value = default; + return false; } - - result = factory(new AssetHandleResolver(this)); } + result = factory(new AssetHandleResolver(this)); + isReady = true; + value = result; return true; } - // TODO: how do we block and wait? -} - -public sealed record ExampleAssets(object A, string B) -{ - public static AssetBundleLoader Load(AssetManager assetManager) - { - var builder = new AssetBundleBuilder(assetManager); - var a = builder.Load(new AssetId("key", "path"), default); - var b = builder.Load(new AssetId("key", "path"), default); - - return builder.Build(r => new ExampleAssets(r.Get(a), r.Get(b))); - } + // TODO: Add a method to block and wait without eating all the CPU. } diff --git a/source/CapriKit.AssetPipeline/AssetCache.cs b/source/CapriKit.AssetPipeline/AssetCache.cs index 4a8fd32..4e0271e 100644 --- a/source/CapriKit.AssetPipeline/AssetCache.cs +++ b/source/CapriKit.AssetPipeline/AssetCache.cs @@ -35,6 +35,13 @@ public TAsset PutOrLease(AssetId id, TAsset candidate) if (Entries.TryGetValue(id, out var entry)) { + // If someone tries to add the same instance twice + // just return it, do not schedule it for dispose + if (object.ReferenceEquals(candidate, entry)) + { + return candidate; + } + PendingDispose.Enqueue(new Entry(id, candidate, 0)); var asset = Cast(entry, id); diff --git a/source/CapriKit.AssetPipeline/AssetDecoder.cs b/source/CapriKit.AssetPipeline/AssetDecoder.cs index 2222bc6..b496ec9 100644 --- a/source/CapriKit.AssetPipeline/AssetDecoder.cs +++ b/source/CapriKit.AssetPipeline/AssetDecoder.cs @@ -9,7 +9,6 @@ namespace CapriKit.AssetPipeline; /// Decodes the generic asset envelope and, using a specialized IAssetTranscoder, the asset itself /// Threading: thread-safe /// -// TODO: AssetDecoder it should be possible to override the output path internal static class AssetDecoder { public static async Task> Decode(AssetId id, IAssetTranscoder decoder, IReadOnlyVirtualFileSystem fileSystem, Stream? inputStreamOverride = default) diff --git a/source/CapriKit.AssetPipeline/AssetEncoder.cs b/source/CapriKit.AssetPipeline/AssetEncoder.cs index 44c73aa..3d52b47 100644 --- a/source/CapriKit.AssetPipeline/AssetEncoder.cs +++ b/source/CapriKit.AssetPipeline/AssetEncoder.cs @@ -12,7 +12,6 @@ namespace CapriKit.AssetPipeline; /// Encodes the generic asset envelope and, using a specialized IAssetTranscoder, the asset itself. /// Threading: thread-safe /// -// TODO: AssetEncoder it should be possible to override the output path internal static class AssetEncoder { public static async Task Encode(AssetId id, IAssetTranscoder encoder, TSettings settings, IVirtualFileSystem fileSystem, Stream? outputStreamOverride = default) @@ -20,18 +19,27 @@ public static async Task Encode(AssetId id, IAssetTranscoder< { ThrowOnFileNotFound(id.Path, fileSystem); var outputPath = ToEncodedFilePath(id); + Stream? output = null; + try + { + output = outputStreamOverride ?? fileSystem.CreateReadWrite(outputPath); + var writer = PipeWriter.Create(output, new StreamPipeWriterOptions(leaveOpen: true)); + var spy = fileSystem.SpyOn(); - using var output = outputStreamOverride ?? fileSystem.CreateReadWrite(outputPath); - var writer = PipeWriter.Create(output); - var spy = fileSystem.SpyOn(); + WriteHeader(writer, encoder); + WriteSettings(writer, encoder, settings); + await WritePayload(writer, id, encoder, settings, spy); + WriteDependencies(writer, spy); - WriteHeader(writer, encoder); - WriteSettings(writer, encoder, settings); - await WritePayload(writer, id, encoder, settings, spy); - WriteDependencies(writer, spy); + await writer.FlushAsync(); + await writer.CompleteAsync(); + } + finally + { + // Only dispose of the stream if we created it. + if (outputStreamOverride is null) { output?.Dispose(); } + } - await writer.FlushAsync(); - await writer.CompleteAsync(); } private static void WriteHeader(PipeWriter writer, IAssetTranscoder encoder) diff --git a/source/CapriKit.AssetPipeline/AssetHandle.cs b/source/CapriKit.AssetPipeline/AssetHandle.cs index 767dbc2..c9d391b 100644 --- a/source/CapriKit.AssetPipeline/AssetHandle.cs +++ b/source/CapriKit.AssetPipeline/AssetHandle.cs @@ -2,12 +2,15 @@ namespace CapriKit.AssetPipeline; -public abstract class AssetHandle() +public abstract class AssetHandle(AssetId id) { - internal protected bool isResolved; - internal protected object? value; + public AssetId Id { get; } = id; - internal AssetBundleLoader? Owner { get; set; } + private object? value; + private volatile bool isResolved; + + + internal AssetBundle? Owner { get; set; } internal bool IsResolved => isResolved; internal object? Value => value; @@ -19,9 +22,9 @@ internal void Resolve(object asset) } } -public sealed class AssetHandle : AssetHandle { } +public sealed class AssetHandle(AssetId id) : AssetHandle(id) { } -public sealed class AssetHandleResolver(AssetBundleLoader owner) +public sealed class AssetHandleResolver(AssetBundle owner) { public TValue Get(AssetHandle promise) { diff --git a/source/CapriKit.AssetPipeline/AssetManager.cs b/source/CapriKit.AssetPipeline/AssetManager.cs index 97641ce..e65ac4e 100644 --- a/source/CapriKit.AssetPipeline/AssetManager.cs +++ b/source/CapriKit.AssetPipeline/AssetManager.cs @@ -47,9 +47,18 @@ public void RegisterTranscoder(IAssetTranscoder + /// Use to defines a bundle of assets to load + /// Threading: thread-safe. + /// + public AssetBundleBuilder CreateBundle() + { + return new AssetBundleBuilder(this); + } + /// /// Starts loading an asset. The asset will either be loaded from the cache, from disk, or rebuild and then loaded. - /// The caller gets a handle to be used in an which can be resolved + /// The caller gets a handle to be used in an which can be resolved /// to the actual asset when loading finishes using /// Threading: thread-safe, can be called from any thread concurrently. This method guarantees that the same asset /// is not loaded multiple times concurrently. @@ -57,7 +66,7 @@ public void RegisterTranscoder(IAssetTranscoder Load(AssetId id, TSettings settings) where TAsset : class { - var handle = new AssetHandle(); + var handle = new AssetHandle(id); // At this time an asset is either already loaded, already requested or requested for the first time. // The lock ensure that this does not change while we check what we should do with the request. @@ -123,10 +132,29 @@ private async Task RequestAsset(AssetId id, TSettings setting } } - // Thread safe - public void Unload(AssetId id) + /// + /// Unloads all assets in the bundle. + /// Threading: Unload updates the internal state of the bundle using a lock so that it is safe + /// to unload the same bundle from multiple threads. + /// + public void Unload(AssetBundle bundle) { - Cache.Return(id); + try + { + RequestLock.Enter(); + if (bundle.IsActive) + { + foreach (var asset in bundle.Assets) + { + Cache.Return(asset); + } + } + } + finally + { + bundle.IsActive = false; + RequestLock.Exit(); + } } /// @@ -203,7 +231,6 @@ private static bool IsUpToDate(IAssetTranscoder /// Used when an asset is loaded and the handler resolved to register the asset with the cache and hot-reloader. /// Thread safe: calling this method from multiple threads, even to materialize the same asset is safe. @@ -215,7 +242,7 @@ private TAsset MaterializeAsset(Asset asse var actualObject = Cache.PutOrLease(asset.Id, asset.Value); var actualWrapper = new Asset(asset.Id, actualObject, asset.BuildMetaData); - HotReloadManager.Track(actualWrapper, transcoder); // TODO: track needs to be thread safe and ignore adding the same thing multiple times! + HotReloadManager.Track(actualWrapper, transcoder); return actualObject; } diff --git a/source/CapriKit.AssetPipeline/HotReloadManager.cs b/source/CapriKit.AssetPipeline/HotReloadManager.cs index c6002d9..eff0862 100644 --- a/source/CapriKit.AssetPipeline/HotReloadManager.cs +++ b/source/CapriKit.AssetPipeline/HotReloadManager.cs @@ -29,7 +29,7 @@ internal sealed partial class HotReloadManager : IDisposable private readonly ConcurrentQueue PendingReloads; private long lastFileChange; - private bool isReloading; + private volatile bool isReloading; public HotReloadManager(ILoggerFactory logger, ScopedFileSystem fileSystem) { @@ -64,6 +64,7 @@ public void Track(Asset asset, IAssetTrans var reloadable = new HotReloadable(asset, transcoder); if (Tracked.TryGetValue(asset.Id, out var assets)) { + // TODO: this is broken!!!! // Prevent adding the exact same instance multiple times if (assets.Any(a => object.ReferenceEquals(a, asset))) { return; } assets.Add(reloadable); @@ -73,19 +74,20 @@ public void Track(Asset asset, IAssetTrans Tracked[asset.Id] = [reloadable]; } - foreach (var dependency in asset.BuildMetaData.Dependencies) - { - var file = dependency.File; - if (Dependents.TryGetValue(file, out var ids)) - { - ids.Add(asset.Id); - } - else - { - ids = [asset.Id]; - Dependents.Add(file, ids); - } - } + RegisterFileDependencies(asset.Id, asset.BuildMetaData.Dependencies); + } + } + + /// + /// Stops tracking an asset + /// + public void UnTrack(AssetId id) + { + // Do not remove all references from dependents as that would require us to go through the entire collection + // but just forgetting it from tracked the asset will no longer be reloaded. + lock (TrackingLock) + { + Tracked.Remove(id); } } @@ -112,20 +114,23 @@ public void Update() /// /// Drains the queue of file events and adds any assets that dependent on this file to PendingRebuilds - /// Threading: Unsafe, must be called single threaded because the Dependents dictionary can only + /// Threading: Unsafe, must be called single threaded because PendingRebuilds can only /// be used by one thread at a time. /// private void DrainFileChanges() { - while (FileChances.TryDequeue(out var @event)) + lock (TrackingLock) { - if (Dependents.TryGetValue(@event.File, out var dependents)) + while (FileChances.TryDequeue(out var @event)) { - lastFileChange = Stopwatch.GetTimestamp(); - foreach (var id in dependents) + if (Dependents.TryGetValue(@event.File, out var dependents)) { - PendingRebuilds.Add(id); - LogPendingReload(Logger, @event.File, id); + lastFileChange = Stopwatch.GetTimestamp(); + foreach (var id in dependents) + { + PendingRebuilds.Add(id); + LogPendingReload(Logger, @event.File, id); + } } } } @@ -143,47 +148,52 @@ private void ReloadOne() var id = PendingRebuilds.First(); PendingRebuilds.Remove(id); - isReloading = true; - - LogReloadStarted(Logger, id); - HotReloadable? target = null; - List candidates; + List? candidates; // Even though this method is single threaded, other methods that allow parallelism can // touch Tracked so we need to put a lock around it. lock (TrackingLock) { - candidates = Tracked[id]; - } + if (!Tracked.TryGetValue(id, out candidates)) + { + return; + } - if (candidates.Count > 0) - { - var pruned = new List(1); + // TODO: we allow users to add the same asset-id multiple times but we expect + // that (after a while) only one of the asset instances is still used by the engine. The + // others will be garbage collected eventually. Still, at this point in time we cannot be sure + // that the first (or last, or..) alive candidate is the one that will survive. + // How should we deal with that? foreach (var candidate in candidates) { if (candidate.IsAlive) { - pruned.Add(candidate); target = candidate; + break; } } } // Not finding a target is normal, it means a file // changed but the asset depending on it is no longer in use. - target?.Reload(FileSystem, PendingReloads) - .FireAndForget(ex => - { - LogReloadFailed(Logger, id, ex); - isReloading = false; - }, - () => - { - LogReloadCompleted(Logger, id); - isReloading = false; - }); - + if (target != null) + { + LogReloadStarted(Logger, id); + + isReloading = true; + Task.Run(() => target.Reload(FileSystem, PendingReloads) + .FireAndForget(ex => + { + LogReloadFailed(Logger, id, ex.SourceException); + isReloading = false; + }, + () => + { + LogReloadCompleted(Logger, id); + isReloading = false; + })); + } } /// @@ -199,6 +209,9 @@ private void HotSwapPending() { LogHotSwapStarted(Logger, action.Id); action.PerformHotSwap(); + + RegisterFileDependencies(action.Id, action.Dependencies); + LogHotSwapCompleted(Logger, action.Id); } catch (Exception ex) @@ -208,6 +221,27 @@ private void HotSwapPending() } } + // Updates the dependents + private void RegisterFileDependencies(AssetId id, IReadOnlyList dependencies) + { + lock (TrackingLock) + { + foreach (var dependency in dependencies) + { + var file = dependency.File; + if (Dependents.TryGetValue(file, out var ids)) + { + ids.Add(id); + } + else + { + ids = [id]; + Dependents.Add(file, ids); + } + } + } + } + public void Dispose() { Watcher.Stop(); @@ -223,7 +257,7 @@ public void Dispose() [LoggerMessage(Level = LogLevel.Information, Message = "Reloading asset pending: {asset}")] private static partial void LogReloadStarted(ILogger logger, AssetId asset); - [LoggerMessage(Level = LogLevel.Error, Message = "Reloading asset completed: {asset}")] + [LoggerMessage(Level = LogLevel.Information, Message = "Reloading asset completed: {asset}")] private static partial void LogReloadCompleted(ILogger logger, AssetId asset); [LoggerMessage(Level = LogLevel.Error, Message = "Reloading asset failed: {asset}")] diff --git a/source/CapriKit.AssetPipeline/HotReloadable.cs b/source/CapriKit.AssetPipeline/HotReloadable.cs index 14891b3..9943f89 100644 --- a/source/CapriKit.AssetPipeline/HotReloadable.cs +++ b/source/CapriKit.AssetPipeline/HotReloadable.cs @@ -3,7 +3,7 @@ namespace CapriKit.AssetPipeline; -internal sealed record HotSwapAction(AssetId Id, Action PerformHotSwap); +internal sealed record HotSwapAction(AssetId Id, IReadOnlyList Dependencies, Action PerformHotSwap); internal abstract class HotReloadable(AssetId id) { @@ -42,6 +42,6 @@ public override async Task Reload(IVirtualFileSystem fileSystem, ConcurrentQueue stream.Seek(0, SeekOrigin.Begin); var hot = await AssetDecoder.Decode(Id, Transcoder, fileSystem, stream); - hotSwapActionQueue.Enqueue(new HotSwapAction(Id, () => Transcoder.HotSwap(cold, hot.Value))); + hotSwapActionQueue.Enqueue(new HotSwapAction(Id, hot.BuildMetaData.Dependencies, () => Transcoder.HotSwap(cold, hot.Value))); } } diff --git a/source/CapriKit.Concurrency/Async/FireAndForgetExtensions.cs b/source/CapriKit.Concurrency/Async/FireAndForgetExtensions.cs index a935aff..26a0f32 100644 --- a/source/CapriKit.Concurrency/Async/FireAndForgetExtensions.cs +++ b/source/CapriKit.Concurrency/Async/FireAndForgetExtensions.cs @@ -1,4 +1,5 @@ using System.Runtime.CompilerServices; +using System.Runtime.ExceptionServices; namespace CapriKit.Concurrency.Async; @@ -11,22 +12,16 @@ public static class FireAndForgetExtensions /// public static void FireAndForget( this Task task, - Action onException, - Action? onCompleted = null, - [CallerMemberName] string member = "", - [CallerFilePath] string file = "", - [CallerLineNumber] int line = 0) + Action onException, + Action? onCompleted = null) { - _ = AwaitAndCatch(task, onException, onCompleted, member, file, line); + _ = AwaitAndCatch(task, onException, onCompleted); } private static async Task AwaitAndCatch( Task task, - Action onException, - Action? onCompleted, - string member, - string file, - int line) + Action onException, + Action? onCompleted) { try { @@ -38,8 +33,9 @@ private static async Task AwaitAndCatch( } catch (Exception ex) { - // The original stack trace is preserved via the inner exception - onException(new FaFTaskException(ex, member, file, line)); + // Preserver the original stack trace and allow the handler to rethrow it. + var capture = ExceptionDispatchInfo.Capture(ex); + onException(capture); } finally { @@ -47,18 +43,3 @@ private static async Task AwaitAndCatch( } } } - -public sealed class FaFTaskException : Exception -{ - public FaFTaskException(Exception innerException, string member, string file, int line) - : base($"A fire-and-forget task started from '{member}' at {file}:{line} failed.", innerException) - { - Member = member; - File = file; - Line = line; - } - - public string Member { get; } - public string File { get; } - public int Line { get; } -} diff --git a/source/CapriKit.Concurrency/Primitives/LightweightChannel.cs b/source/CapriKit.Concurrency/Primitives/LightweightChannel.cs index 71c56c1..6b98cc5 100644 --- a/source/CapriKit.Concurrency/Primitives/LightweightChannel.cs +++ b/source/CapriKit.Concurrency/Primitives/LightweightChannel.cs @@ -8,10 +8,8 @@ namespace CapriKit.Concurrency.Primitives; /// Lightweight variant of that allows ONE reader /// to receive items from MULTIPLE writers. Has the following semantics: /// - void Write(T value) always succeeds and supports multiple writers working in parallel -/// - void Write(Exception exception) always succeeds but only keeps the first exception -/// - bool TryRead(out T? value) is non blocking, it returns one item from the internal queue if available -/// if there are no items but an exception was written, it rethrows the exception. -/// +/// - void Write(ExceptionDispatchInfo exception) always succeeds and supports multiple writers working in parallel +/// - bool TryRead(out T? value) is non blocking, it first drains the exceptions in the queue (one per call) then the items. /// Note that there being no items doesn't mean the work is done. Users must keep track of the number of items /// they expect to see if the work is done, there is no Completed method or completion tracking to avoid /// problems like 'write after complete' that require extensive locking. @@ -19,12 +17,11 @@ namespace CapriKit.Concurrency.Primitives; public sealed class LightweightChannel where T : notnull { private readonly ConcurrentQueue Queue; - private volatile ExceptionDispatchInfo? Error; - + private readonly ConcurrentQueue Errors; public LightweightChannel() { Queue = []; - Error = null; + Errors = []; } public void Write(T value) @@ -32,20 +29,23 @@ public void Write(T value) Queue.Enqueue(value); } - public void Write(Exception exception) + public void Write(ExceptionDispatchInfo exception) { - var error = ExceptionDispatchInfo.Capture(exception); - Interlocked.CompareExchange(ref Error, error, null); + Errors.Enqueue(exception); } public bool TryRead([NotNullWhen(true)] out T? value) { + if (Errors.TryDequeue(out var error)) + { + error.Throw(); + } + if (Queue.TryDequeue(out value)) { return true; } - Error?.Throw(); return false; } } diff --git a/source/CapriKit.Tests/AssetPipeline/AssetManagerTests.cs b/source/CapriKit.Tests/AssetPipeline/AssetManagerTests.cs index 316fe7f..42db527 100644 --- a/source/CapriKit.Tests/AssetPipeline/AssetManagerTests.cs +++ b/source/CapriKit.Tests/AssetPipeline/AssetManagerTests.cs @@ -41,9 +41,9 @@ public async Task LoadAsset() var transcoder = new TestTranscoder(); assetManager.RegisterTranscoder(transcoder); - var id = new AssetId(string.Empty, AssetFile); + var id = new AssetId(AssetFile); - var builder = new AssetBundleBuilder(assetManager); + var builder = assetManager.CreateBundle(); var handle = builder.Load(id, default); var loader = builder.Build(resolver => new TestBundle(resolver.Get(handle))); @@ -57,6 +57,22 @@ await Assert.That(() => await Assert.That(bundle).IsNotNull(); await Assert.That(bundle.Text).IsEqualTo(TranscoderText); + + // Load again to verify loading the same thing twice gives us the cached value + var altBuilder = new AssetBundleBuilder(assetManager); + var altHandle = altBuilder.Load(id, default); + var altLoader = altBuilder.Build(resolver => new TestBundle(resolver.Get(altHandle))); + + TestBundle? altBundle = null; + await Assert.That(() => + { + assetManager.Update(); + return altLoader.IsReady(out altBundle); + }) + .Eventually(v => v.IsTrue(), TimeSpan.FromSeconds(5)); + + await Assert.That(altBundle).IsNotNull(); + await Assert.That(altBundle.Text).IsSameReferenceAs(bundle.Text); } private record TestBundle(string Text); diff --git a/source/CapriKit.Tests/Concurrency/Async/FireAndForgetExtensionsTests.cs b/source/CapriKit.Tests/Concurrency/Async/FireAndForgetExtensionsTests.cs index 85a1a0a..4c7dcd4 100644 --- a/source/CapriKit.Tests/Concurrency/Async/FireAndForgetExtensionsTests.cs +++ b/source/CapriKit.Tests/Concurrency/Async/FireAndForgetExtensionsTests.cs @@ -1,4 +1,5 @@ using CapriKit.Concurrency.Async; +using System.Runtime.ExceptionServices; namespace CapriKit.Tests.Concurrency.Async; @@ -31,7 +32,7 @@ public async Task FireAndForget_AlreadyCompletedTask() [Test] public async Task FireAndForget_CancelledTask() { - Exception? observed = null; + ExceptionDispatchInfo? observed = null; var completed = new TaskCompletionSource(); var cancelled = Task.FromCanceled(new CancellationToken(canceled: true)); @@ -44,14 +45,14 @@ public async Task FireAndForget_CancelledTask() [Test] public async Task FireAndForget_FaultedTask() { - Exception? observed = null; + ExceptionDispatchInfo? observed = null; var completed = new TaskCompletionSource(); var faulted = Task.FromException(new InvalidOperationException()); faulted.FireAndForget(ex => observed = ex, () => completed.SetResult()); await completed.Task.WaitAsync(Timeout); - await Assert.That(observed).IsTypeOf(); - await Assert.That(observed!.InnerException).IsTypeOf(); + await Assert.That(observed).IsNotNull(); + await Assert.That(observed.SourceException).IsTypeOf(); } } diff --git a/source/CapriKit.Tests/Concurrency/Primitives/LightweightChannelTests.cs b/source/CapriKit.Tests/Concurrency/Primitives/LightweightChannelTests.cs index 6b19fa3..d0e2a8f 100644 --- a/source/CapriKit.Tests/Concurrency/Primitives/LightweightChannelTests.cs +++ b/source/CapriKit.Tests/Concurrency/Primitives/LightweightChannelTests.cs @@ -1,4 +1,5 @@ using CapriKit.Concurrency.Primitives; +using System.Runtime.ExceptionServices; namespace CapriKit.Tests.Concurrency.Primitives; @@ -32,35 +33,16 @@ public async Task TryRead_ReturnsFalseWhenChannelIsEmpty() } [Test] - public async Task TryRead_RethrowsTheWrittenException() + public async Task TryRead_RethrowsTheWrittenExceptionBeforeReturningOtherItems() { var channel = new LightweightChannel(); - channel.Write(new InvalidOperationException()); + channel.Write(4); + channel.Write(ExceptionDispatchInfo.Capture(new InvalidOperationException())); await Assert.That(() => channel.TryRead(out _)).Throws(); - } - - [Test] - public async Task TryRead_DrainsQueuedItemsBeforeThrowingTheException() - { - var channel = new LightweightChannel(); - channel.Write(new InvalidOperationException()); - channel.Write(42); var read = channel.TryRead(out var value); - await Assert.That(read).IsTrue(); - await Assert.That(value).IsEqualTo(42); - await Assert.That(() => channel.TryRead(out _)).Throws(); - } - - [Test] - public async Task Write_KeepsOnlyTheFirstException() - { - var channel = new LightweightChannel(); - channel.Write(new InvalidOperationException()); - channel.Write(new FormatException()); - - await Assert.That(() => channel.TryRead(out _)).Throws(); + await Assert.That(value).IsEqualTo(4); } } From 2535eae5202223d6a1352655665975b20335298e Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Sat, 15 Aug 2026 11:51:55 +0200 Subject: [PATCH 41/53] More review --- .../AssetPipelineVNextReviewContinued.html | 696 ++++++++++++++++++ ...nseToFindingsOfAssetPipelineVNextReview.md | 11 + 2 files changed, 707 insertions(+) create mode 100644 Research/AssetPipelineVNextReviewContinued.html diff --git a/Research/AssetPipelineVNextReviewContinued.html b/Research/AssetPipelineVNextReviewContinued.html new file mode 100644 index 0000000..017570b --- /dev/null +++ b/Research/AssetPipelineVNextReviewContinued.html @@ -0,0 +1,696 @@ + + + + + +Asset Pipeline vNext — Follow-up + + + + +
+
+
CapriKit · feature/asset_pipeline · c7517e7
+

Asset Pipeline vNext — Follow-up

+

Verification of the fixes made in response to + AssetPipelineVNextReview.html, plus answers to the four design questions in + ResponseToFindingsOfAssetPipelineVNextReview.md. Items that are fixed get one line; + everything else is reported in full.

+
+ Verified against 11 findings closed, 3 still open + Method read + 7 executed repros + Date 2026-08-15 + Working tree left clean, no source modified +
+ +
+
Solution build
PASS
0 warnings, 0 errors
+
Hot reload
WORKS
first time, end‑to‑end
+
Closed
11
B1 B2 B4 H1 H2 H4 M2 M3 M4 M8 + L1 L4
+
Still open
3
B3 critical, M1+M6 high
+
New
4
N1–N4, all small
+
Answered
4
H3+H5, M1, M8, L5
+
+
+
+ +
+
+

The headline

+

The two things the last report said had never run — the staleness fast path and + hot reload — both now run. That is the big result, and it changes what the rest of the plan should look like.

+ +
+ Executed, not inferred + 
[B2] first run  ready=True  encodes=1
+[B2] second run ready=True  SECOND-RUN ENCODES = 0   <-- loaded from the .cka
+
+[H]  initial load ready=True  text='Hello World'
+[H]  hot reload #1 swapped=True after 517ms  text='Goodbye World'
+[H]  hot reload #2 swapped=True after 532ms  text='Third Text'
+

Two consecutive reloads is the important half of the second result: it proves the isReloading + latch really clears, which is what H2 was about. 517 ms is the 500 ms debounce plus one frame.

+
+ +

Against that, one critical survives. B3 is not fixed — the asset handed to a bundle is still + disposed on the very frame it resolves, and the guard added to prevent it contains the same type-confusion typo as H5. + The new per-bundle Unload is a good idea whose lease accounting does not balance, which is the M1 answer.

+ +
+ One consequence worth flagging +

The previous report argued that restructuring the envelope (S2) was low-risk because B2 proved the staleness + path had never executed. That argument is now void — the path is live and load-bearing. If you still want S2, + it now needs the second-run test to exist first.

+
+
+
+ +
+
+

Closed — one line each

+ +
+ + + + + + + + + + + + + + + + + + + +
IDVerifiedResult
B1✓ fixedAssetManagerExtensions.cs was deleted rather than reshaped; dotnet build CapriKit.slnx now succeeds with 0 warnings and 0 errors across all 16 projects.
B2✓ fixedThe else at AssetManager.cs:121 closes the fall-through; a second manager over a directory that already has an up-to-date .cka performs 0 encodes and no longer throws KeyNotFoundException.
B4✓ fixedLightweightChannel's sticky error slot became a ConcurrentQueue<ExceptionDispatchInfo> drained one per read: a missing file makes Update() throw exactly 1 of 40 frames with the real FileNotFoundException, and healthy assets still load afterwards. Matches your intent of "fail loudly once". See N4 for a small hazard the new overload pair introduces.
H1✓ fixedStreamPipeWriterOptions(leaveOpen: true) plus the conditional dispose in finally; the encode→seek→decode round trip in HotReloadable.Reload completes. The mirror-image bug in AssetDecoder is still there — N3.
H2✓ fixedisReloading is volatile, and the assignment moved inside if (target != null) so the null-target case cannot latch. Two consecutive edits both hot-swapped, ~520 ms each.
H4✓ fixedThe candidate iteration in ReloadOne and the whole of DrainFileChanges now sit inside TrackingLock. One residual: PendingRebuilds is mutated inside the lock at :131 and outside it at :148–149. Harmless while both are main-thread — and the M8 answer below deletes the lock entirely, which resolves it.
M2✓ fixedA dedicated bool isReady replaces the result == null test, so struct bundles are materialised once.
M3✓ fixedvolatile bool isResolved written after value and read before Value gives a correct release/acquire pair. Reader side is IsReady → resolver.Get, which respects that order.
M4✓ fixedReloadOne wraps the reload in Task.Run, so the synchronous prefix of Reload (and any D3DCompile) is off the main thread.
M8✓ fixedHotSwapAction now carries hot.BuildMetaData.Dependencies and HotSwapPending calls RegisterFileDependencies, so a new #include is registered without a restart. Note it only ever adds — a removed #include keeps triggering rebuilds until restart. Cosmetic; mentioning it so it is not rediscovered.
L1✓ fixedSuccessful reload logs at Information.
L4✓ fixed*.cka is in .gitignore.
L6✓ partExampleAssets is gone from the shipped surface. AssetHandle's public constructor still lets external code forge a handle; low priority, listed for completeness.
M7waivedPer your note: transcoders are registered at start-up. Worth one sentence of XML doc on RegisterTranscoder saying so, since the failure mode is a worker-thread throw.
M5carriedStill open, as you asked. HotReloadManager.Dispose clears its four collections without TrackingLock and does not wait for an in-flight reload; AssetManager.Dispose drains neither Incoming nor the outstanding requests. Disposing while a load is in flight is undefined today.
+
+
+
+ +
+
+

Still open

+ +
+
B3Not fixed — the bundle still receives an asset that was disposed on the same frame
+
AssetManager.cs:171-181 (materializer still called per handle) · AssetCache.cs:40 (the new guard compares the wrong operands)
+
repro executedcorrectnessthreading
+ +

Two things had to change and neither landed. The drain still materialises once per handle:

+ 
var handles = Outstanding[id];
+foreach (var handle in handles)
+{
+    var asset = materializer();   // still once PER HANDLE
+    handle.Resolve(asset);
+}
+ +

and the guard that was added to PutOrLease to make the second call harmless compares a + TAsset against an Entry wrapper — two unrelated types, so it is always + false, exactly like the H5 guard it was modelled on:

+ 
if (object.ReferenceEquals(candidate, entry))   // TAsset  vs  Entry
+{
+    return candidate;
+}
+//                                  entry.Asset is what you meant
+ + Failure scenario — executed + 
[B3-one-bundle]  ready=True same=True A.IsDisposed=True B.IsDisposed=True
+[B3-two-bundles] ready=True same=True IsDisposed=True
+

Both handles resolve to the same instance and that instance is already disposed when the bundle is handed over, + because PutOrLease queued the live object into PendingDispose and Cache.Collect() + — three lines later in the same Update() — disposed it. No exception is raised. For a + VertexShaderTranscoder that is a released ID3D11VertexShader handed to gameplay code.

+ + Fix — four small edits, and they also settle M1 + 
// 1. AssetCache.PutOrLease: compare against the stored instance
+if (ReferenceEquals(candidate, entry.Asset)) { return candidate; }
+
+// 2. AssetCache: one place to add the leases for the other bundles
+public void AddLeases(AssetId id, int count)
+{
+    if (count <= 0) { return; }
+    lock (Lock)
+    {
+        ObjectDisposedException.ThrowIf(isDisposed, this);
+        if (Entries.TryGetValue(id, out var entry)) { entry.RefCount += count; }
+    }
+}
+
+// 3. AssetBundleBuilder: one handle per id per bundle
+private readonly Dictionary<AssetId, AssetHandle> Handles = [];
+
+public AssetHandle<TAsset> Load<TAsset, TSettings>(AssetId id, TSettings settings)
+    where TAsset : class
+{
+    if (Handles.TryGetValue(id, out var existing))
+    {
+        return existing as AssetHandle<TAsset>
+            ?? throw new InvalidOperationException($"{id} is already claimed in this bundle as another type");
+    }
+
+    var handle = assetManager.Load<TAsset, TSettings>(id, settings);
+    Handles.Add(id, handle);
+    return handle;
+}
+
+// 4. AssetManager.Update: materialise once, lease once per waiting bundle
+if (!Outstanding.Remove(id, out var handles)) { continue; }
+
+var asset = materializer();                 // exactly once
+Cache.AddLeases(id, handles.Count - 1);     // step 3 makes this the bundle count
+foreach (var handle in handles) { handle.Resolve(asset); }
+

Step 3 is what makes step 4 correct rather than approximately correct: once a builder can only produce one handle per + id, every handle in Outstanding[id] necessarily belongs to a different bundle, so + handles.Count is the number of bundles that will each call Return once.

+
+ +
+
M1+M6The per-bundle Unload is the right unit, but the accounting does not balance and an in-flight unload corrupts the bundle
+
AssetManager.cs:140-158 · AssetBundle.cs:39-44 · AssetCache.cs:103
+
repro executedcorrectnessthreading
+ +

This is the direct answer to "is this mechanism sound and thread-safe?" — the mutual exclusion is fine, + the arithmetic is not. Three separate problems.

+ + 1 — leases are counted per handle, returns per distinct id +

AssetBundle.Assets is a HashSet<AssetId>, so Unload calls + Return once per distinct id. But leases are taken once per handle — + by Cache.TryLease on the cache-hit path and by the repeated PutOrLease on the load path. + The two only agree when every bundle holds exactly one handle per id:

+ 
[B3-one-bundle] Unload ok
+  Dispose threw Exception: Cache will leak 1 entries that have not
+  been returned before the cache was disposed.
+

The invariant you want, and it is worth writing it into the XML doc of both Return and + PutOrLease, is: RefCount(id) equals the number of active bundles that contain + id. B3's steps 3 and 4 above establish exactly that on the load path; the cache-hit path in + Load gets it for free from step 3.

+ + 2 — unloading an in-flight bundle throws, and then poisons the bundle + 
[M6] Unload while in flight threw InvalidOperationException:
+     Returned AssetId { Path = Hello.txt, Key =  } which was not found in the cache.
+[M6] after load completes: ready=True IsDisposed=False
+[M6] second Unload: ok            <-- silently did nothing
+[M6] Dispose threw Exception: Cache will leak 1 entries ...
+

One unhandled case produces four compounding failures: Return throws for an id that has not been + materialised yet; the loop aborts so the remaining ids are never returned either; the finally + sets IsActive = false anyway (N1), so the second Unload is a no-op and the leak is now + permanent; and the load completes afterwards and takes a lease nobody can ever return.

+

This is V2's open cancellation question arriving in code. There are two honest answers:

+
    +
  • Minimal — refuse it. Check Outstanding first and throw + before mutating anything, leaving IsActive true so a later Unload still works. + Three lines, no corruption, decision deferred. + 
    lock (RequestLock)
+{
+    if (!bundle.IsActive) { return; }
+    foreach (var id in bundle.Assets)
+    {
+        if (Outstanding.ContainsKey(id))
+        {
+            throw new InvalidOperationException(
+                $"Cannot unload a bundle while {id} is still loading. Unload after IsReady.");
+        }
+    }
+    // ... only now mutate
+}
    • +
  • Proper — let a dead owner cancel its own lease. Unload marks the bundle + inactive and skips ids that are still in Outstanding; the drain then counts only the handles whose + owner is still active, and if that count is zero it returns the single lease PutOrLease just took. + This needs handle.Owner to be known at Load time, which it is not today — + Owner is assigned in Build(). The structural fix is to let + CreateBundle() hand out the object that is the bundle identity, with + Build<TBundle> attaching the factory and returning a typed view over it. That also makes + "unload before Build" legal, which it currently is not.
    • +
+ + 3 — two small mechanical issues in the method itself (N1, N2) +

See the next section.

+ +

Thread-safety, separately: holding RequestLock across the whole of Unload + is correct — it makes Unload mutually exclusive with Load and with the + Update() drain, which is the guarantee you need. The lock ordering stays consistent + (RequestLock → Cache.Lock, never the reverse). Two things are unguarded but + benign today: AssetBundle.AssetSet is written by Build on one thread and read by + Unload on another with no barrier, and IsReady keeps handing out its cached + result after the bundle has been unloaded, so gameplay code can still reach disposed assets through a + loader it kept.

+
+
+
+ +
+
+

New issues

+

All four are small. N1 and N2 are inside the new Unload; N3 and N4 came in with the H1 and + B4 fixes.

+ +
+ + + + + + + + +
IDSeverityFindingLocation
N1HighUnload's finally sets bundle.IsActive = false even when the loop threw partway through, so a bundle that failed to return its leases can never be unloaded again. Set it inside the body, after the returns succeed.AssetManager.cs:155
N2LowRequestLock.Enter() is inside the try, so if Enter ever throws, the finally calls Exit() without holding the lock and a SynchronizationLockException masks the real error. Either move Enter() above the try, or just use lock (RequestLock) { ... } — the whole method fits.AssetManager.cs:142-157
N3LowH1 was fixed in AssetEncoder but not in its mirror: using var input = inputStreamOverride ?? fileSystem.OpenRead(...) still disposes a stream the caller owns. Benign today only because HotReloadable passes a MemoryStream, whose Dispose is idempotent. Apply the same finally shape so the two sides stay symmetric.AssetDecoder.cs:23
N4LowThe new Write(T) / Write(ExceptionDispatchInfo) overload pair silently misroutes when T is ExceptionDispatchInfo: the non-generic overload wins, so a value written as a payload lands in the error queue and is rethrown at the reader. Verified: Ch<EDI>.Write(edi) -> queue=0, errors=1. Renaming one side to WriteError removes the whole class of confusion and reads better at the call site in AssetManager.Load.LightweightChannel.cs:27,32
+
+ +
+ Adjacent, outside this review's scope — JobResult<T>.Match is broken for value types +

Not part of the asset pipeline (only CapriKit.Tests.Tool uses it), but it sits next to the code you + changed for B4, so: for a value-type T, Result is default(T) rather than + null, so a failed result calls both callbacks:

+ 
JobResult<int>.Failure("job", edi).Match(onSuccess, onFailure)
+  onSuccess called with 0   <-- should not happen
+  onFailure called
+

The Debug.Assert(Result == null ^ Exception == null) in the constructor fires for the same reason, so in + a Debug build this trips before it misbehaves. The usual fix is to store the discriminator explicitly + (private readonly bool isSuccess;) rather than inferring it from nullness, and make Match an + if/else so the two arms are exclusive by construction.

+
+
+
+ +
+
+

Your questions

+ + +
+

H3 / H5 — how should HotReloadManager and AssetCache work together? + Can the cache be the authority?

+
+

Yes, and it collapses both findings into one change. The bug underneath H3 and H5 is the same: + HotReloadManager is trying to answer a question it does not own. WeakReference.IsAlive + answers "has the GC collected this?", and the dedupe guard tries to answer "is this the instance we + already track?". The cache already answers both, exactly and without GC timing: it holds one entry per + AssetId, it knows the winning instance, and it knows the moment the last lease goes away.

+ +

H3 is confirmed live, by the way — a hot swap runs happily against an unloaded, disposed asset:

+ 
[H3] after Unload: IsDisposed=True text='Hello World'
+[H3] hot swapped a DISPOSED/unloaded asset = True (IsDisposed=True)
+ + Step 1 — the cache reports eviction; it already knows + 
// AssetCache: true when the last lease went away
+public bool Return(AssetId id)
+{
+    lock (Lock)
+    {
+        ObjectDisposedException.ThrowIf(isDisposed, this);
+        if (!Entries.TryGetValue(id, out var entry)) { return false; }
+
+        entry.RefCount--;
+        if (entry.RefCount > 0) { return false; }
+
+        Entries.Remove(id);
+        PendingDispose.Enqueue(entry);
+        return true;
+    }
+}
+
+// AssetManager.Unload: eviction is now the one place tracking ends
+if (Cache.Return(id)) { HotReloadManager.UnTrack(id); }
+

You already wrote UnTrack — nothing calls it. This is its caller.

+ + Step 2 — HotReloadable holds a strong reference, and Tracked stops being a list + 
private readonly TAsset Instance;   // was WeakReference<TAsset>
+// IsAlive, the candidate loop and both TODOs are deleted
+
+private readonly Dictionary<AssetId, HotReloadable> Tracked = [];
+
+public void Track<TAsset, TSettings>(Asset<TAsset, TSettings> asset, IAssetTranscoder<TAsset, TSettings> transcoder)
+    where TAsset : class
+    => Tracked[asset.Id] = new HotReloadable<TAsset, TSettings>(asset, transcoder);
+

H5 does not need fixing — it stops existing. The list was only there because the manager + could not tell which of several instances was the live one. The cache guarantees there is exactly one: + PutOrLease returns the winner and the loser is discarded. With one entry per id there is nothing to + dedupe, no candidate to choose, and no pruned list to forget to assign. The lifetime of the strong + reference is now bounded by UnTrack instead of by the GC, which is the whole point.

+ + Step 3 — close the last-mile race +

One window remains: a rebuild is already on the pool when the bundle is unloaded. The swap must be validated on the + main thread, at the moment it is applied. Carry the target on the action so it can be compared:

+ 
internal sealed record HotSwapAction(
+    AssetId Id, object Target, IReadOnlyList<Dependency> Dependencies,
+    Action PerformHotSwap, Action DiscardNewParts);
+
+// AssetCache
+public bool IsCurrent(AssetId id, object instance)
+{
+    lock (Lock) { return Entries.TryGetValue(id, out var e) && ReferenceEquals(e.Asset, instance); }
+}
+
+// HotSwapPending, main thread
+if (!Cache.IsCurrent(action.Id, action.Target))
+{
+    action.DiscardNewParts();   // nothing installed it; do not leak the rebuilt asset
+    continue;
+}
+

DiscardNewParts matters: HotSwap is what normally absorbs or frees + newParts, so a skipped swap leaks it otherwise.

+ +

This depends on an ordering that is already correct in Update() — + Cache.Collect() at AssetManager.cs:184 runs before + HotReloadManager.Update() at :185. Evict-then-swap is what makes a stale swap detectable + rather than a race. Worth a one-line comment saying so, because reordering those two lines would silently reopen H3.

+ +
+ + + + + + + + + +
TodayWith the cache as authority
liveness meansWeakReference.IsAlive — GC timingthe cache still holds a lease — deterministic
Tracked valueList<HotReloadable> + candidate loop + 2 TODOsone HotReloadable
H5 dedupebroken ReferenceEquals across two typesnot needed
swap onto a dead assethappens (repro'd above)rejected by IsCurrent
size—roughly −35 lines
+
+
+
+ + +
+

M8 — there is a lot of locking in HotReloadManager. Can it be simpler or more explicit? + Hot reload is rare, so should it use concurrent collections more?

+
+

Go the other way: delete the lock, do not add concurrent collections. Map the state first — + once you do, the reason the locking feels heavy is that it is guarding against a caller that does not exist.

+ +
+ + + + + + + + + + + +
StateWritten byRead byGuard actually needed
Trackedmain — Track ← MaterializeAsset ← Update()mainnone
Dependentsmain — Track and HotSwapPendingmainnone
PendingRebuildsmainmainnone
lastFileChangemainmainnone
FileChanceswatcher threadmainalready a concurrent queue
PendingReloadspoolmainConcurrentQueue ✓
isReloadingmain sets, pool clearsmainvolatile ✓
+
+ +

Only the last three rows are genuinely cross-thread, and all three are already handled. TrackingLock + guards the first four — which never leave the main thread, because Track is reached only from + MaterializeAsset, which is reached only from the Update() drain. And that is not an + accident of the current code: materialisation has to happen on the main thread, because that is the only + place a transcoder's ID3D11DeviceContext work can legally run. It is a structural constraint, so it is + safe to lean on.

+ +

So delete TrackingLock — four lock blocks — and replace it with something + that states the constraint instead of defending against its opposite:

+ 
// AssetManager captures this once, in its constructor
+private readonly int MainThreadId = Environment.CurrentManagedThreadId;
+
+[Conditional("DEBUG")]
+private void AssertMainThread([CallerMemberName] string caller = "")
+    => Debug.Assert(Environment.CurrentManagedThreadId == MainThreadId,
+        $"{caller} must run on the thread that created the AssetManager.");
+

Call it at the top of Update, Collect, Track, UnTrack and + IsReady. It costs nothing in Release, it fires immediately and by name if the assumption ever breaks, + and it removes the contradiction the last report flagged — Track documented as + "thread-safe, multiple threads" while DrainFileChanges two methods later says "must be called single + threaded". This is also step 7 of the previous report's suggested order, done cheaply.

+ +
+ Why not concurrent collections +

Because they would be less safe while looking safer, and this class has already been bitten by exactly + that. ConcurrentDictionary<AssetId, List<HotReloadable>> makes the dictionary thread-safe + and leaves the inner List unguarded — that is H4's original shape, verbatim. And + Track needs lookup, mutate and RegisterFileDependencies to be atomic together; + concurrent collections give per-operation atomicity, never multi-step. Rarity is an argument for a plain lock over + a clever one, not for a lock-free type.

+
+ +

Two side effects worth having: the H4 residual disappears (PendingRebuilds stops being inside the lock + in one place and outside it in another), and so does the current reliance on System.Threading.Lock + being reentrant — Track holds TrackingLock and calls + RegisterFileDependencies, which takes it again. That works, but it is a subtlety you no longer have to + know about. Do the H3 change first: with Tracked holding one entry instead of a list, there is barely + anything left for a lock to protect.

+
+
+ + +
+

L5 — where am I missing ConfigureAwait(false), given that + FireAndForget already sets it?

+
+

All twelve awaits in the pipeline, in four files. FireAndForget's + ConfigureAwait(false) applies only to its own await of the outer task — it does not + propagate into the awaits inside RequestAsset or anything it calls. Each of these captures the ambient + SynchronizationContext independently.

+ +
+ + + + + + + + +
FileLinesWhat is awaited
AssetManager.cs114, 117, 128, 129TryDecodeBuildMetaData, Decode ×2, Encode
AssetEncoder.cs31, 34, 35, 64WritePayload, FlushAsync, CompleteAsync, encoder.Encode
AssetDecoder.cs29, 65input.ReadExactlyAsync ×2
HotReloadable.cs40, 43AssetEncoder.Encode, AssetDecoder.Decode
+
+ +

None of them is a live bug today, and it is worth being clear about why, because it also tells you + when it stops being true. Every one of these paths is entered through Task.Run — + Load at AssetManager.cs:92 and, since the M4 fix, ReloadOne at + HotReloadManager.cs:185. On a pool thread SynchronizationContext.Current is + null, so every continuation already resumes on the pool. The M4 fix is what removed the one path that + could have hit this.

+ +

It becomes load-bearing the moment any of these methods is awaited directly from a thread that has a + context — a WinForms or WPF host, or a custom game-loop synchronisation context. Since these ship as a NuGet + package you do not get to know the host. So: add it to all twelve, and rather than maintaining it by hand, turn on + CA2007 for the library projects so the compiler keeps it honest. CapriKit.IO already does + this by hand, which is exactly the sort of consistency that decays silently.

+
+
+
+
+ +
+
+

Suggested order from here

+

Much shorter than last time. Steps 1 and 2 are the same edit set, which is the main reason to do them together.

+
    +
  1. B3 + M1 — the four edits. +

    Fix the PutOrLease operand, add AddLeases, dedupe in the builder, materialise once. This + is one coherent change that establishes a single invariant: RefCount(id) is the number of active + bundles holding id. Write that invariant in the XML docs while it is fresh.

    2. +
  2. M6 — pick minimal or proper, then N1 and N2. +

    If you take the minimal route, N1 becomes moot because nothing throws mid-loop any more. If you take the proper + route, do the CreateBundle() restructure first, since that is what makes Owner known at + Load time.

    3. +
  3. H3 + H5 — make the cache the authority. +

    Return returns bool, Unload calls UnTrack, strong reference, + Tracked becomes a single entry, IsCurrent guards the swap. Net negative lines, and it + deletes both TODOs in HotReloadManager.

    4. +
  4. M8 — delete TrackingLock, add AssertMainThread. +

    Cheapest after step 3, because there is much less state left to reason about.

    5. +
  5. Turn the repros into tests. +

    The seven listed at the bottom of this report are the paths AssetManagerTests still cannot reach. + The second-run test in particular is now protecting real behaviour rather than dead code.

    6. +
  6. Then N3, N4, L5, M5 — and only then revisit S1–S7. +

    Re-read S2's risk note before starting it: its justification was that the staleness path had never executed, + which is no longer true.

    7. +
+
+
+ +
+
+

Method. Read of the full pipeline at c7517e7 plus the changed + CapriKit.Concurrency files, a full-solution build, and seven repros executed as temporary TUnit tests + against the real AssetManager: second-run staleness (B2), two handles in one bundle and in two bundles + (B3), unload-in-flight (M6), failed load followed by 40 frames (B4), hot reload twice in a row (H1/H2), and hot swap + after unload (H3). Terminal output quoted above is verbatim. The temporary test file and a scratch console project used + to confirm N4 were removed; the working tree is clean and no source file was modified.

+

Prior art. Continues AssetPipelineVNextReview.html and answers + ResponseToFindingsOfAssetPipelineVNextReview.md. Findings S0–S7 and P1–P8 from the previous + report are untouched and still stand, except that S2's risk assessment is now out of date.

+
+
+ + + diff --git a/Research/ResponseToFindingsOfAssetPipelineVNextReview.md b/Research/ResponseToFindingsOfAssetPipelineVNextReview.md index ac62630..c051c9d 100644 --- a/Research/ResponseToFindingsOfAssetPipelineVNextReview.md +++ b/Research/ResponseToFindingsOfAssetPipelineVNextReview.md @@ -1,3 +1,9 @@ +I have worked through the findings you reported in C:\projects\csharp\CapriKit\research\AssetPipelineVNextReview.html. Read that first so that you are familiar with the issues and issue numbering. I have made substational changes to CapriKit.AssetPipelile and few changes to some tests and CapriKit.Concurrency to address most of the issues. + +Below are what I worked on and the questions I have for you. Write a new report html report (call it continued or something like that) and answer my questions. In your new report only give a short one line answer if the problem was fixed satisfactory. If the problem still exists or has created a new problem, report it as normal. + +--- + I made changes to fix issues B1-B4, please verify. Note that for B4 I do not mind if one failed load kills the game, but I now added a way to handle that so that the game can at least throw a complete error message. I made changes to fix issues H1, H2 and H4, please verify @@ -16,3 +22,8 @@ M7: Ignore that for now, users are supposed to initialize the asset manager with M8: I think I fxed this, but there is now a lot of locking going on in HotReloadManager, can we make this simpler or at least more explicit. Hot reloading is very rare so maybe its better to use the concurrent collection types more often? + + +L5: in which places am I missing `ConfigureAwait(false)` not that .FireAndForget sets `ConfigureAwait(false)` + +I have not looked at S0 to S7 From 7789b54a83e95fafcff85eba8729801e63b114cf Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Sat, 15 Aug 2026 14:18:05 +0200 Subject: [PATCH 42/53] WIP --- ...pelineHotReloadWithoutAssetReferences.html | 692 ++++++++++++++++++ .../HotReloadManagerV2.cs | 228 ++++++ 2 files changed, 920 insertions(+) create mode 100644 Research/AssetPipelineHotReloadWithoutAssetReferences.html create mode 100644 source/CapriKit.AssetPipeline/HotReloadManagerV2.cs diff --git a/Research/AssetPipelineHotReloadWithoutAssetReferences.html b/Research/AssetPipelineHotReloadWithoutAssetReferences.html new file mode 100644 index 0000000..d00079a --- /dev/null +++ b/Research/AssetPipelineHotReloadWithoutAssetReferences.html @@ -0,0 +1,692 @@ + + + + + +Hot Reload Without Asset References + + + + +
+
+
CapriKit · feature/asset_pipeline · design exploration
+

Hot Reload Without Asset References

+

What happens to the code, and to the number of things you have to keep in your head, if + HotReloadManager stops holding assets entirely — and instead asks AssetCache for a + lease whenever a file changes.

+
+ Baseline 321 lines — HotReloadManager.cs + HotReloadable.cs + Status exploration, nothing implemented + Date 2026-08-15 +
+ +
+
Concepts removed
7
two added
+
Findings closed
H3 H5
by construction, not by fix
+
Lines
−27
321 → ~294, only −8%
+
New locks
0
all cache calls main-thread
+
Gets harder
M5
shutdown with a live lease
+
Verdict
Do it
certainty, not size
+
+
+
+ +
+
+

The idea, and why it works

+ +

The contract changes from a sentence with a hedge in it to a sentence without one.

+ +
+
+

Today

+

“HotReloadManager keeps weak references to the assets it has + seen, and when a file changes it picks the first one the GC has not collected yet and hopes that is the one the + engine is actually using.”

+
+
+

Proposed

+

“HotReloadManager knows how to rebuild an asset. + AssetCache knows whether anyone still wants it. When a file changes, the manager asks the + cache for a lease; if it gets one, it rebuilds.”

+
+
+ +

The whole design rests on one property that is already true in the cache you have written, and it is worth being + precise about it because everything else follows:

+ + 
// AssetCache.Return, line 93-99
+entry.RefCount--;
+// Evict items immediately, but only dispose of them in Collect
+if (entry.RefCount <= 0)
+{
+    Entries.Remove(id);          // <-- gone from Entries the instant the last user lets go
+    PendingDispose.Enqueue(entry);
+}
+ +

Eviction from Entries is immediate and deterministic; only the Dispose call + is deferred to Collect(). So TryLease returning false means exactly + “no live user holds this asset” — not “the GC happens to have got round to it”. That is + the question HotReloadable.IsAlive has been trying and failing to answer since it was written, and the + cache has been answering it correctly all along. Nobody was asking.

+ +
+ The two guarantees a lease buys you +

1. Liveness. TryLease succeeds ⇒ at least one real user holds this asset. Entries + at refcount 0 are not in Entries, so they cannot be leased. H3 — hot-swapping an asset + Collect already disposed — becomes unrepresentable rather than merely unlikely.

+

2. Identity stability. While you hold the lease the entry cannot be evicted, and + PutOrLease only ever inserts when an entry is absent. So the instance you leased is still the + instance the cache holds when your rebuild lands one frame later. That is a proof, not a hope — and it deletes + the whole stale-swap detection layer (IsCurrent, HotSwapAction.Target, + DiscardNewParts) that the previous report proposed for exactly this window.

+
+
+
+ +
+
+

What the manager still has to hold

+ +

This is where the idea meets the type system, and it is the one place where the result is less tidy than + the sentence above suggests.

+ +

You cannot delete the per-asset object. On a file-change event the manager has an AssetId and nothing + else — no TAsset, no TSettings. But Cache.TryLease<TAsset> is + generic, and AssetEncoder.Encode needs both type parameters. Something has to carry those types from + registration time to reload time, and in C# that something is an object with an abstract base:

+ + 
internal abstract class ReloadRecipe(AssetId id)
+{
+    public AssetId Id { get; } = id;
+    public IReadOnlyList<Dependency> Dependencies { get; set; } = [];
+
+    /// <summary>
+    /// Leases the live instance from the cache and starts a rebuild on the thread pool.
+    /// Returns false when no live instance exists, which means this recipe is dead and can be forgotten.
+    /// The lease is released in HotSwapPending, exactly once, whether the rebuild succeeds or fails.
+    /// </summary>
+    public abstract bool TryStartReload(AssetCache cache, IVirtualFileSystem fileSystem,
+        ConcurrentQueue<ReloadResult> results, out Task reload);
+}
+ +

So the class hierarchy survives. What changes is what it holds, and that is the entire point:

+ +
+
+

HotReloadable<TAsset, TSettings> — today

+ 
private readonly WeakReference<TAsset> Instance;   // <-- an asset
+private readonly TSettings Settings;
+private readonly IAssetTranscoder<TAsset, TSettings> Transcoder;
+
+public override bool IsAlive => Instance.TryGetTarget(out var _);
+
+
+

ReloadRecipe<TAsset, TSettings> — proposed

+ 
private readonly TSettings Settings;
+private readonly IAssetTranscoder<TAsset, TSettings> Transcoder;
+
+// no asset field, so no IsAlive, and nothing to keep alive
+
+
+ +

One field goes. Everything that field's ambiguity forced goes with it, and that is a much longer list than + one field: IsAlive on the base and the override, the candidate-selection loop and its five-line TODO in + ReloadOne, the List<HotReloadable> that only existed because the manager could not tell + the instances apart, the broken dedupe guard (H5), the pruned list that gets built and discarded, and + UnTrack, which has no caller today and now never needs one.

+ +
+ Make it structural, not documentary +

“Holds no asset references” should be visible in the signature rather than true by inspection. Today + Track takes the whole Asset<TAsset, TSettings> and simply chooses not to keep + .Value. Narrow the parameters instead, and the property becomes something the compiler enforces:

+ 
// before: the asset is right there, one careless line away from being captured
+public void Track<TAsset, TSettings>(Asset<TAsset, TSettings> asset, IAssetTranscoder<TAsset, TSettings> transcoder)
+
+// after: there is nothing to capture
+public void Track<TAsset, TSettings>(AssetId id, TSettings settings,
+    IReadOnlyList<Dependency> dependencies, IAssetTranscoder<TAsset, TSettings> transcoder)
+

Same reasoning as AssetHandle having no .Value accessor: the invariant that survives is + the one the type system carries.

+
+
+
+ +
+
+

The code, before and after

+ +

Track — 22 lines to 6

+
+
+

HotReloadManager.cs:58-79

+ 
lock (TrackingLock)
+{
+    var reloadable = new HotReloadable<TAsset, TSettings>(asset, transcoder);
+    if (Tracked.TryGetValue(asset.Id, out var assets))
+    {
+        // TODO: this is broken!!!!
+        // Prevent adding the exact same instance multiple times
+        if (assets.Any(a => object.ReferenceEquals(a, asset))) { return; }
+        assets.Add(reloadable);
+    }
+    else
+    {
+        Tracked[asset.Id] = [reloadable];
+    }
+
+    RegisterFileDependencies(asset.Id, asset.BuildMetaData.Dependencies);
+}
+
+
+

proposed

+ 
// One live instance per id is the cache's guarantee,
+// so one recipe per id is all we can ever need.
+Tracked[id] = new ReloadRecipe<TAsset, TSettings>(id, settings, transcoder)
+{
+    Dependencies = dependencies
+};
+
+ReplaceFileDependencies(id, dependencies);
+
+
+

The dedupe question that H5 got wrong does not get a correct answer here — it stops being asked. There is one + entry per id because the cache guarantees one instance per id, and re-tracking is an overwrite of a recipe that was + equal anyway.

+ +

ReloadOne — 54 lines to 24, and the TODO goes

+
+
+

HotReloadManager.cs:144-197

+ 
lock (TrackingLock)
+{
+    if (!Tracked.TryGetValue(id, out candidates)) { return; }
+
+    // TODO: we allow users to add the same asset-id multiple times but we expect
+    // that (after a while) only one of the asset instances is still used by the engine. The
+    // others will be garbage collected eventually. Still, at this point in time we cannot be sure
+    // that the first (or last, or..) alive candidate is the one that will survive.
+    // How should we deal with that?
+    foreach (var candidate in candidates)
+    {
+        if (candidate.IsAlive) { target = candidate; break; }
+    }
+}
+
+// Not finding a target is normal, it means a file
+// changed but the asset depending on it is no longer in use.
+if (target != null) { /* snip: dispatch */ }
+
+
+

proposed

+ 
if (!Tracked.TryGetValue(id, out var recipe)) { return; }
+
+// The cache is the authority. A successful lease answers
+// "is anyone still using this?" and guarantees the instance
+// survives until the swap lands.
+if (!recipe.TryStartReload(Cache, FileSystem, PendingReloads, out var reload))
+{
+    Forget(id);   // nobody holds it: drop the recipe and its file edges
+    return;
+}
+
+LogReloadStarted(Logger, id);
+isReloading = true;
+reload.FireAndForget(
+    ex => { LogReloadFailed(Logger, id, ex.SourceException); isReloading = false; },
+    () => { isReloading = false; });
+
+
+ +

The recipe — where the lease is taken

+ 
public override bool TryStartReload(AssetCache cache, IVirtualFileSystem fileSystem,
+    ConcurrentQueue<ReloadResult> results, out Task reload)
+{
+    // Main thread. Fails when the last bundle already returned the asset,
+    // which is exactly the case where reloading would be pointless or unsafe.
+    if (!cache.TryLease<TAsset>(Id, out var cold))
+    {
+        reload = Task.CompletedTask;
+        return false;
+    }
+
+    reload = Task.Run(async () =>
+    {
+        try
+        {
+            using var stream = new MemoryStream();
+            await AssetEncoder.Encode(Id, Transcoder, Settings, fileSystem, stream).ConfigureAwait(false);
+
+            stream.Seek(0, SeekOrigin.Begin);
+            var hot = await AssetDecoder.Decode(Id, Transcoder, fileSystem, stream).ConfigureAwait(false);
+
+            results.Enqueue(new ReloadResult(Id, hot.BuildMetaData.Dependencies,
+                () => Transcoder.HotSwap(cold, hot.Value)));
+        }
+        catch
+        {
+            // Always enqueue: HotSwapPending is the single place the lease is released.
+            results.Enqueue(ReloadResult.Failed(Id));
+            throw;
+        }
+    });
+
+    return true;
+}
+ +

HotSwapPending — the single release point

+ 
private void HotSwapPending()
+{
+    while (PendingReloads.TryDequeue(out var result))
+    {
+        try
+        {
+            if (result.PerformHotSwap is not null)
+            {
+                LogHotSwapStarted(Logger, result.Id);
+                result.PerformHotSwap();
+                ReplaceFileDependencies(result.Id, result.Dependencies!);
+                LogHotSwapCompleted(Logger, result.Id);
+            }
+        }
+        catch (Exception ex)
+        {
+            LogHotSwapFailed(Logger, result.Id, ex);
+        }
+        finally
+        {
+            Cache.Return(result.Id);   // the lease from TryStartReload, released exactly once
+        }
+    }
+}
+

One finally, one release point, on the same thread that took the lease. That is the whole lifetime story + and it fits on a screen — which is the property to protect if you build this.

+
+
+ +
+
+

The clarity ledger

+

The honest measure of this change is not lines. It is how many uncertain questions the code stops + containing.

+ +
+ + + + + + + + + + + + + +
Question the code has to answerTodayAfter
Is this asset still in use?WeakReference.IsAlive — GC timing, wrong answer possible (H3)TryLease — deterministic, by definition
Which of N tracked instances is the live one?unanswerable; a 5-line TODO picks the firstquestion deleted — there is one
Is this asset already tracked?broken ReferenceEquals across two types (H5)question deleted — one entry per id
Will the instance survive until the swap?hopethe lease guarantees it
Who tells hot reload that an asset died?nobody — UnTrack has no callernobody needs to; the next lease just fails
Is the tracking table still accurate?grows forever; pruned is computed and discardedpruned on the failed lease
Could the swap target be stale?yes — needs IsCurrent + Target + DiscardNewPartsno — the lease makes it impossible
Who owns the lease, and when is it released?— no lease existsnew: the recipe takes it, HotSwapPending releases it
Does hot reload know about the cache?no — independent objectsnew: yes, a constructor dependency
+
+ +

Seven uncertain things out, two certain things in. The two that come in are both of the kind you can + point at in a code review: an ownership rule with one release point, and an edge in the object graph. The seven that + go out are the kind you can only find by reasoning about GC timing, and one of them (H3) is a confirmed defect that + was reproduced last week.

+ +
+ A structural note about the file-change map +

Dependents stays — a file→ids map is build-graph data, not an asset reference, and this + design does not touch it. But it gains a property it does not have today: it becomes self-pruning. + When TryLease fails, the manager has learned something definite ("this asset is gone") and can drop the + recipe and its file edges on the spot. That is what Forget(id) does above, and it is why the + pruned list can finally be deleted instead of fixed — the code never had a trustworthy signal to + prune on before.

+
+
+
+ +
+
+

What this costs

+ +
+ The real trade +

You are exchanging GC ambiguity for an explicit lifetime obligation that crosses a thread boundary and + a queue. A WeakReference has no obligations: forget about it and nothing bad happens. A lease has + exactly one, and forgetting it is silent until shutdown. That is a good trade — the obligation is local, + checkable, and lives in one finally — but it is a real one and it is worth naming before you start.

+
+ +
+ + + + + + + +
New obligationWhere it livesIf you get it wrong
Release the lease exactly onceHotSwapPending's finallyRefcount never reaches 0; the asset is never disposed; Cache.Dispose throws “will leak N entries”
Enqueue a result even when the rebuild throwsthe recipe's catch { enqueue; throw; }Same leak, and only on the error path — the one you will not exercise by hand
Drain PendingReloads at shutdownHotReloadManager.DisposeM5 gets worse: a reload in flight at shutdown holds a lease no one will release
+
+ +

Two other things get slightly harder

+
    +
  • M5 moves from "undefined" to "actively broken". Today, disposing with a reload in flight is merely + unspecified. With leases, an in-flight reload holds a refcount, so AssetCache.Dispose will report a leak + that is not a real leak. If you build this, M5 stops being a "later" item — at minimum + HotReloadManager.Dispose must drain PendingReloads and return those leases before + Cache.Dispose runs.
    • +
  • The refcount invariant grows a second kind of holder. The invariant proposed for B3/M1 was + “RefCount(id) equals the number of active bundles holding id”. It + becomes “… plus at most one in-flight reload”. Arguably that is cleaner phrased the other + way round — a refcount counts claims, and a reload is a claim — but you have to restate it, and + the XML docs on Return and PutOrLease should say so.
    • +
+ +

And one thing that looks like a cost but is not

+

A reload can now finish for an asset the game unloaded halfway through: the lease keeps the entry alive, the swap + runs, then the lease is returned and the asset is evicted and disposed. That is wasted work — one rebuild of + something nobody wants — but it is correct, and it replaces today's behaviour, which is a hot swap onto + an object Collect has already disposed. Wasted work in a dev-only code path on a debounced file change is + not worth engineering away.

+
+
+ +
+
+

Size, honestly

+

This is not a size win. Report it to yourself as a certainty win and you will not be disappointed.

+ +
+ + + + + + + + + + + + +
PieceNowAfterWhy
Track228no list, no dedupe, no TODO
ReloadOne5424candidate loop and its TODO deleted
UnTrack → Forget98same size, finally has a caller
HotSwapPending1921+ the release finally
dependency map1928only if you take the exact-replacement upgrade
HotReloadable → ReloadRecipe4751−weak ref, −IsAlive, +lease and failure enqueue
cache field + ctor03the new edge
Total321~294−27 lines, −8%
+
+ +

For comparison, S6 from the first review — hot reload via closures, opt-in behind a flag so shipping builds + never construct it — was −155 lines. These are orthogonal, and this one makes S6 easier: once the + recipe has no asset field it is pure behaviour, which is exactly what a closure is good at. A closure that captures a + transcoder and settings is fine; a closure that captures an asset is the bug you are removing. Do this first, and S6 + becomes a mechanical rewrite instead of a redesign.

+ +
+ Zero new locking +

Both cache calls happen on the main thread — TryLease in ReloadOne, + Return in HotSwapPending, both reached from AssetManager.Update() and, notably, + outside RequestLock (AssetManager.cs:184-185). So hot reload takes + Cache.Lock and nothing else, no nesting, and the existing + RequestLock → Cache.Lock ordering is untouched. The recommendation from the + previous report to delete TrackingLock and replace it with a debug AssertMainThread is + independent of this change and composes with it — after both, there is very little left for a lock to protect.

+
+ +

One invariant also relaxes: the previous report asked you to comment that Cache.Collect() must + run before HotReloadManager.Update(), because evict-then-swap was what made a stale swap detectable. With + leases a stale swap cannot happen at all, so that ordering stops being load-bearing. One fewer thing that breaks + silently if someone reorders two lines.

+
+
+ +
+
+

A bonus that falls out

+

Your phrasing — a file that an asset might depend on — is precise, and this design + is what makes the “might” cheap to remove.

+ +

Dependents is an over-approximation today because RegisterFileDependencies only ever + adds. An .hlsl that drops an #include keeps that stale edge until process restart, + so touching the removed file triggers a full rebuild forever. With consultation the false positive is cheap when the + asset is dead — one failed TryLease — but it still costs a real rebuild when the asset is + live.

+ +

The fix is available now and was not before: a rebuild produces the complete fresh dependency set, and the + recipe can hold the previous one (it is a list of paths, not an asset). So the add becomes a replace:

+ + 
private void ReplaceFileDependencies(AssetId id, IReadOnlyList<Dependency> fresh)
+{
+    if (Tracked.TryGetValue(id, out var recipe))
+    {
+        foreach (var old in recipe.Dependencies)
+        {
+            if (Dependents.TryGetValue(old.File, out var ids) && ids.Remove(id) && ids.Count == 0)
+            {
+                Dependents.Remove(old.File);
+            }
+        }
+
+        recipe.Dependencies = fresh;
+    }
+
+    foreach (var dependency in fresh)
+    {
+        // snip: existing add
+    }
+}
+ +

About nine extra lines, and Dependents becomes exact instead of monotone. That also closes the residual + noted against M8 in the follow-up report. It is optional — the design works without it — but it is the + cheapest it will ever be, because the recipe already exists and already has nowhere dangerous to point.

+
+
+ +
+
+

Roads not taken

+ +
+ Put the recipe inside AssetCache.Entry — rejected +

The maximal version of the idea: if the cache holds the recipe too, HotReloadManager keeps only + Dependents and its two queues. Tempting, and it would delete Tracked entirely.

+

Rejected because it destroys the thing that makes the cache useful as an authority in the first + place: it is dumb, and it is about lifetime only. Give it transcoder references and it becomes a registry of + everything known about an asset, which means every future question ("what settings? what dependencies? what + version?") has an obvious wrong home. The object? Tag variant is worse still — untyped and + spooky-at-a-distance. Keep the cache answering exactly one question: does anyone still hold this?

+
+ +
+ Derive the transcoder from the cached instance's runtime type — does not work +

It would be neat if the manager could skip recipes entirely: take entry.Asset.GetType() and look up the + transcoder in AssetManager.Transcoders. Two reasons it fails, both worth knowing so it does not get + re-proposed. Transcoders is keyed on the declared type parameter (IVertexShader), + not the runtime type (VertexShader), so the lookup misses. And settings are per-request and typed, so + even a successful lookup would not tell you what to rebuild with. The recipe object is doing real work; it + is the existential that carries <TAsset, TSettings> across an untyped event.

+
+ +
+ Keep weak references, add an eviction callback — strictly worse +

This was the previous report's H3 recommendation: have the cache call UnTrack(id) on eviction, keep the + weak reference, and guard the swap with IsCurrent. It works, and it is roughly the same number of lines + (−35 versus −27). But it keeps all three uncertain concepts alive — GC timing, candidate selection, + stale-swap detection — and adds callback plumbing on top. The lease approach removes the questions instead of + answering them. Prefer this design; treat the earlier H3 advice as superseded.

+
+
+
+ +
+
+

Verdict

+ +
+

Build it — but for the right reason

+

At −27 lines this will not feel like a simplification while you are typing it. It is worth doing because of + what it deletes from the problem rather than from the file: seven questions the current code either answers + wrongly or cannot answer at all, replaced by two it answers by construction. H3 and H5 stop being findings to fix and + become states that cannot be reached, and the two TODOs in HotReloadManager — including the one + that just says this is broken!!!! — are deleted rather than resolved.

+

It also lands at the right time. Hot reload works end to end as of this week, so for the first time there is a + working baseline to compare against: build it, then re-run the two-consecutive-reloads check and the + swap-after-unload check. The second one should flip from swapped=True (today's bug) to + swapped=False, and that single assertion is the whole design in one line.

+

Order: finish B3 + M1 first — the lease arithmetic has to be right before you add a second + kind of lease holder. Then this. Then M5, which this change promotes from "later" to "required". Then M8's + TrackingLock deletion, which by that point is nearly free.

+
+
+
+ +
+
+

Scope. A design exploration, not a review — nothing here is implemented and no code was + changed. Line counts are estimates against the current HotReloadManager.cs (274 lines) and + HotReloadable.cs (47 lines) at c7517e7. The one load-bearing claim — that + TryLease fails deterministically once the last user returns an asset — is read directly from + AssetCache.Return at lines 93–99, where Entries.Remove happens immediately and only the + Dispose is deferred to Collect.

+

Relationship to the other reports. Supersedes the H3 recommendation in + AssetPipelineVNextReviewContinued.html (cache-as-authority via eviction callback), which reached a similar + place by a worse route. Independent of, and compatible with, the B3/M1 lease-accounting fix and the M8 + TrackingLock deletion in that same report, and it makes S6 from + AssetPipelineVNextReview.html easier rather than harder.

+
+
+ + + diff --git a/source/CapriKit.AssetPipeline/HotReloadManagerV2.cs b/source/CapriKit.AssetPipeline/HotReloadManagerV2.cs new file mode 100644 index 0000000..fc2e5a9 --- /dev/null +++ b/source/CapriKit.AssetPipeline/HotReloadManagerV2.cs @@ -0,0 +1,228 @@ +using CapriKit.Concurrency.Async; +using CapriKit.IO; +using CapriKit.IO.Watchers; +using Microsoft.Extensions.Logging; +using System.Collections.Concurrent; +using System.Diagnostics; + +namespace CapriKit.AssetPipeline; + +internal record ReloadResult(AssetId Id, IReadOnlyList NewDependencies, Action HotSwap); + +internal abstract record ReloadableV2 +{ + public abstract Task Reload(ConcurrentQueue resultQueue); +} + +internal record ReloadableV2(AssetId Id, AssetBuildMetaData Metadata, IAssetTranscoder Transcoder, IVirtualFileSystem FileSystem, AssetCache Cache) + : ReloadableV2 + where TAsset : class +{ + public override async Task Reload(ConcurrentQueue resultQueue) + { + if (Cache.TryLease(Id, out var cold)) + { + try + { + using var steam = new MemoryStream(); + // We store the encoded asset in memory instead of on disk to prevent + // touching the file while other threads are also working on it. + using var stream = new MemoryStream(); + await AssetEncoder.Encode(Id, Transcoder, Metadata.Settings, FileSystem, stream); + + stream.Seek(0, SeekOrigin.Begin); + var hot = await AssetDecoder.Decode(Id, Transcoder, FileSystem, stream); + + resultQueue.Enqueue(new ReloadResult(Id, hot.BuildMetaData.Dependencies, + () => + { + Transcoder.HotSwap(cold, hot.Value); + Cache.Return(Id); + })); + } + catch + { + Cache.Return(Id); + throw; + } + } + } +} + +internal sealed partial class HotReloadManagerV2 : IDisposable +{ + private static readonly TimeSpan MinWaitTime = TimeSpan.FromSeconds(0.5); + private readonly ILogger Logger; + private readonly AssetCache Cache; + private readonly ScopedFileSystem FileSystem; + private readonly IVirtualFileSystemWatcher Watcher; + private readonly FileSystemEventQueue FileChanges; + private readonly Dictionary Tracked; + private readonly Dictionary> Dependents; + + private readonly HashSet PendingRebuilds; + private readonly ConcurrentQueue PendingReloads; + + private readonly Lock Lock; + + private long lastFileChange; + + public HotReloadManagerV2(ILoggerFactory logger, AssetCache cache, ScopedFileSystem fileSystem) + { + Logger = logger.CreateLogger(); + Cache = cache; + FileSystem = fileSystem; + Lock = new(); + Tracked = []; + Dependents = []; + PendingRebuilds = []; + PendingReloads = []; + + Watcher = FileSystem.Watch(); + FileChanges = new(Watcher); + } + + + + public void Track(AssetId id, AssetBuildMetaData metadata, IAssetTranscoder transcoder) + where TAsset : class + { + lock (Lock) + { + // Track each asset only once + if (Tracked.TryGetValue(id, out var _)) { return; } + Tracked[id] = new ReloadableV2(id, metadata, transcoder, FileSystem, Cache); + + foreach (var dependency in metadata.Dependencies) + { + var file = dependency.File; + if (Dependents.TryGetValue(file, out var ids)) + { + ids.Add(id); + } + else + { + ids = [id]; + Dependents.Add(file, ids); + } + } + } + } + + public void Update() + { + DrainFileChanges(); + var elapsed = Stopwatch.GetElapsedTime(lastFileChange); + if (elapsed > MinWaitTime) + { + Rebuild(); + } + + HotSwapPending(); + } + + + /// + /// Drains the queue of file events and adds any assets that dependent on this file to PendingRebuilds + /// Threading: thread-safe + /// + private void DrainFileChanges() + { + lock (Lock) + { + while (FileChanges.TryDequeue(out var @event)) + { + if (Dependents.TryGetValue(@event.File, out var dependents)) + { + lastFileChange = Stopwatch.GetTimestamp(); + foreach (var id in dependents) + { + PendingRebuilds.Add(id); + LogPendingReload(Logger, @event.File, id); + } + } + } + } + } + + /// + /// Starts rebuilding the asset in the set + /// Threading: thread-safe + /// + private void Rebuild() + { + lock (Lock) + { + foreach (var id in PendingRebuilds) + { + if (Tracked.TryGetValue(id, out var reloadable)) + { + Task.Run(() => + { + LogReloadStarted(Logger, id); + reloadable.Reload(PendingReloads); // TODO: do I FireAndForget the outer, inner or both? + LogReloadCompleted(Logger, id); + }).FireAndForget(ex => + { + LogReloadFailed(Logger, id, ex.SourceException); + // TODO: consider error scenarios, especially + // what happens to the lease? + }); + } + } + PendingRebuilds.Clear(); + } + } + + /// + /// Hot swaps the assets that have been reloaded. + /// Threading: Unsafe, the contract from used here + /// requires that assets are only hot swapped on the main thread + /// + private void HotSwapPending() + { + while (PendingReloads.TryDequeue(out var reloadable)) + { + try + { + LogHotSwapStarted(Logger, reloadable.Id); + reloadable.HotSwap(); + + RegisterFileDependencies(reloadable.Id, reloadable.NewDependencies); + + LogHotSwapCompleted(Logger, reloadable.Id); + } + catch (Exception ex) + { + LogHotSwapFailed(Logger, reloadable.Id, ex); + } + } + } + + public void Dispose() + { + Watcher.Stop(); + // TODO: drain in-progress reloads + } + + [LoggerMessage(Level = LogLevel.Information, Message = "Detected file change: {path}, affecting asset: {asset}")] + private static partial void LogPendingReload(ILogger logger, FilePath path, AssetId asset); + + [LoggerMessage(Level = LogLevel.Information, Message = "Reloading asset pending: {asset}")] + private static partial void LogReloadStarted(ILogger logger, AssetId asset); + + [LoggerMessage(Level = LogLevel.Information, Message = "Reloading asset completed: {asset}")] + private static partial void LogReloadCompleted(ILogger logger, AssetId asset); + + [LoggerMessage(Level = LogLevel.Error, Message = "Reloading asset failed: {asset}")] + private static partial void LogReloadFailed(ILogger logger, AssetId asset, Exception exception); + + [LoggerMessage(Level = LogLevel.Information, Message = "Hot-swapping asset started: {asset}")] + private static partial void LogHotSwapStarted(ILogger logger, AssetId asset); + + [LoggerMessage(Level = LogLevel.Information, Message = "Hot-swapping asset completed: {asset}")] + private static partial void LogHotSwapCompleted(ILogger logger, AssetId asset); + + [LoggerMessage(Level = LogLevel.Error, Message = "Hot-swapping asset failed: {asset}")] + private static partial void LogHotSwapFailed(ILogger logger, AssetId asset, Exception exception); +} From 6fddf69e193147b7611a5605153c76153a48ba74 Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Mon, 17 Aug 2026 00:04:00 +0200 Subject: [PATCH 43/53] WIP2 --- Research/HowIDidIt.md | 164 ++++++++++ .../HotReloadManagerV2.cs | 4 +- .../HotReloadManagerV3.cs | 308 ++++++++++++++++++ .../HotReloadPipeline.cs | 77 +++++ source/CapriKit.AssetPipeline/TrackedAsset.cs | 79 +++++ .../AssetPipeline/HotReloadManagerV3Tests.cs | 123 +++++++ 6 files changed, 754 insertions(+), 1 deletion(-) create mode 100644 Research/HowIDidIt.md create mode 100644 source/CapriKit.AssetPipeline/HotReloadManagerV3.cs create mode 100644 source/CapriKit.AssetPipeline/HotReloadPipeline.cs create mode 100644 source/CapriKit.AssetPipeline/TrackedAsset.cs create mode 100644 source/CapriKit.Tests/AssetPipeline/HotReloadManagerV3Tests.cs diff --git a/Research/HowIDidIt.md b/Research/HowIDidIt.md new file mode 100644 index 0000000..585289f --- /dev/null +++ b/Research/HowIDidIt.md @@ -0,0 +1,164 @@ +# How I did it: HotReloadManagerV3 + +_Context: we were finishing the asset pipeline on the `feature/asset_pipeline` branch. Loading assets already +worked end-to-end, hot-reloading did not. Three earlier attempts (`HotReloadManager`, `HotReloadManagerV2`, +`HotReloadPipeline`) were abandoned. This document explains the choices behind the fourth attempt._ + +## The shape of the problem + +Hot-reloading is a small state machine that is awkward because its three steps have different threading rules: + +| Step | Where it must run | Why | +| --- | --- | --- | +| Notice a file changed | any thread (the watcher's) | the OS decides when | +| Rebuild + reload | any thread | it is slow, the main thread must not stall | +| Hot-swap | **main thread only** | `IAssetTranscoder.HotSwap` says so (it touches GPU resources) | + +So the object has to hop threads twice, and along the way we must never leave a lease behind in the +`AssetCache`, never dangle a reference, and never let a failure damage an asset that is already loaded. + +## The central decision: the main thread owns all the state + +The single idea that made this version simpler than the previous three is that **only the main thread owns +mutable state machine state**. `Update()` is the only place where work advances a step: + +``` +Update() + ├─ MarkStaleAssets() drain the file event queue → which assets are stale? + ├─ StartReloads() lease + Task.Run per stale asset → thread pool does the slow part + └─ FinishReloads() for every completed task: hot-swap, then return the lease +``` + +The background tasks are pure: they take an input, produce a `ReloadedAsset`, and touch nothing else. All the +bookkeeping (which asset is stale, which rebuild is running, which lease is held) lives in main-thread-only +fields. That is why there is only one lock in the class, and it guards only the two collections that `Track` +(any thread) genuinely shares with `Update`. + +The earlier attempts inverted this: the background task pushed results into a concurrent queue *and* returned +its own lease *and* was responsible for its own error handling. That spread the ownership of a lease across +two threads, which is exactly where V2's `// TODO: what happens to the lease?` came from. + +## Why I did not serialize to one asset at a time + +You offered to let me handle one asset at a time and suspected it would make things *more* complex. It would. +Running rebuilds in parallel is the naturally simple option here: + +- **Parallel** needs one dictionary, `InFlight: AssetId → Task`. Starting work is + `InFlight.Add(id, task)`, finishing it is "is the task completed?". +- **Serial** needs that *plus* a queue of assets waiting for their turn, plus a "am I currently busy?" flag, + plus a rule for what to do when the file of a queued asset changes again while it waits. + +Serial only removes concurrency I never had to reason about anyway, because the tasks share nothing. + +## The lease protocol + +The cache is the authority on whether an asset is still alive, so a rebuild has to hold a lease for its whole +duration. The invariant I settled on is: + +> **`TryStartReload` takes exactly one lease, and `FinishReload` returns exactly one lease, in a `finally`.** + +To make that hold, the lease is taken **synchronously on the main thread before the task is started**, not +inside the task: + +```csharp +// TrackedAsset +if (!cache.TryLease(Id, out var live)) { reload = null; return false; } +reload = Task.Run(() => Reload(live, fileSystem)); +``` + +This buys two things. The manager always knows whether a lease exists (no "did the task get one before it +threw?" question), and the task receives the live instance as a plain argument, so it never has to look at +shared state. A failed `TryLease` is not an error, it is how we learn that nobody uses the asset anymore, +which is also the moment we stop tracking it. + +Because `AssetCache.Return` takes only an `AssetId`, the manager can return the lease without knowing the +asset's type. That is what keeps the type erasure cheap. + +## Type erasure + +As you predicted in the shell, this needs an abstract non-generic base plus a generic subclass: + +- `TrackedAsset` — `AssetId` + `Dependencies` + `TryStartReload(...)`. This is what lives in the dictionaries. +- `TrackedAsset` — adds the `TSettings` and the `IAssetTranscoder`, and + is the only place that knows the real types. +- `ReloadedAsset` — the non-generic result: the new dependency list plus an `Action` that performs the swap. + The generic types are captured inside that closure, so the main thread can run the swap without knowing them. + +I deliberately did **not** store the whole `AssetBuildMetaData` on the tracked asset. Only `Settings` and +`Dependencies` are ever used again; the transcoder id and version are already fixed by the transcoder instance. + +## Debouncing + +An editor saving a file produces several change events (truncate, write, flush). Rebuilding on the first one +means reading a half-written file. So a rebuild only starts when nothing relevant changed for +`Debounce` (default 0.5s). + +Two details worth noting: + +- The timer is only reset by changes to files that some tracked asset actually depends on. Otherwise the asset + pipeline writing its own `.cka` build files would keep postponing rebuilds forever. +- The window is global rather than per-asset. Saving file A delays a pending rebuild of unrelated asset B by + half a second. For a development-only feature that is invisible, and a per-asset timestamp would mean another + dictionary. + +The constructor takes an optional `debounce` so tests can pass `TimeSpan.Zero` and stay deterministic instead +of sleeping. `Update` compares with `>=` precisely so that zero means "no debounce" and never depends on a +timer tick landing. + +## Failure handling + +The guarantee is that a failure never invalidates a live asset. That falls out of the design almost for free, +because nothing is mutated until the very last step: + +- **Rebuild/reload fails** → the task faults, we log it, the live asset was never touched. Its old contents + stay correct and the next file change tries again. +- **Hot-swap fails** → the transcoder made whatever it made of the instance; we can only log it. Throwing here + would take the game down over a development feature, which is the wrong trade. +- **Either way** → the `finally` returns the lease, so a broken shader can never leak cache entries. + +The rebuild writes into a `MemoryStream` instead of over the `.cka` file on disk, because another thread may +be reading that exact file to load the same asset. The cost is that the on-disk build stays stale after a hot +reload, so the next startup rebuilds the asset once. That seemed clearly better than corrupting a concurrent load. + +## Dispose + +`Dispose` stops the watcher, drops everything not yet started, and then **waits for the running rebuilds and +finishes them through the normal path** rather than abandoning them. That is not just tidiness: an abandoned +rebuild holds a lease (the cache would report leaked entries) and owns a freshly built asset whose native +resources are only cleaned up by `HotSwap`. Draining and completing them normally handles both, and reuses +`FinishReloads` verbatim. + +## Locking + +There is one lock, and the rules are: + +1. It guards `Tracked` and `Dependents` only — the two collections `Track` shares with `Update`. +2. **The hot-swap runs outside of it.** `HotSwap` is user code that may load another asset, which would take + the asset manager's `RequestLock` and then this lock. Holding this lock during the swap would be a real + deadlock, not a theoretical one. +3. Lock order in the system is `AssetManager.RequestLock → HotReloadManager.Lock → AssetCache.Lock`, and + nothing acquires them in the other direction, so there is no cycle. + +## Things I noticed while doing this + +Two of these will bite when you wire V3 into `AssetManager`: + +1. **`AssetManager.Dispose` disposes in the wrong order.** It calls `Cache.Dispose()` *before* + `HotReloadManager.Dispose()`. Since the hot reload manager can hold leases, the cache will throw + "will leak N entries", and the manager's `Cache.Return` will then throw `ObjectDisposedException`. The hot + reload manager must be disposed first. (`HotReloadManagerV3Tests` declares `cache` before `sut` so that C#'s + reverse disposal order gets this right, and the leak check then doubles as a lease-accounting assertion.) +2. **`AssetDecoder.Decode` checks that the `.cka` file exists even when you pass a stream override.** It works + today only because a loaded asset always has a build on disk. It is a trap for any future in-memory-only path. +3. **`HotReloadManagerV2.cs` did not compile** — it calls `RegisterFileDependencies`, which was never written. + That is committed in `HEAD`, so the branch did not build. I commented the call out with a `TODO` so I could + verify V3; deleting the abandoned experiments is your call. + +## What I left out + +- No cancellation of in-flight rebuilds. Saving a file twice quickly just rebuilds twice; the second result + wins because the swaps are ordered by the main thread. +- No coalescing of a rebuild with a concurrent first-time load of the same asset. The lease makes it safe, only + wasteful, and it is rare. +- Dead tracked assets are only cleaned up when a file change reveals that the cache dropped them. An asset that + is unloaded and never touched again leaves a small entry behind until then. diff --git a/source/CapriKit.AssetPipeline/HotReloadManagerV2.cs b/source/CapriKit.AssetPipeline/HotReloadManagerV2.cs index fc2e5a9..6a55907 100644 --- a/source/CapriKit.AssetPipeline/HotReloadManagerV2.cs +++ b/source/CapriKit.AssetPipeline/HotReloadManagerV2.cs @@ -188,7 +188,9 @@ private void HotSwapPending() LogHotSwapStarted(Logger, reloadable.Id); reloadable.HotSwap(); - RegisterFileDependencies(reloadable.Id, reloadable.NewDependencies); + // TODO: this abandoned experiment does not compile, RegisterFileDependencies was never written. + // Commented out so that the project builds, see HotReloadManagerV3 for the version that works. + //RegisterFileDependencies(reloadable.Id, reloadable.NewDependencies); LogHotSwapCompleted(Logger, reloadable.Id); } diff --git a/source/CapriKit.AssetPipeline/HotReloadManagerV3.cs b/source/CapriKit.AssetPipeline/HotReloadManagerV3.cs new file mode 100644 index 0000000..dea2312 --- /dev/null +++ b/source/CapriKit.AssetPipeline/HotReloadManagerV3.cs @@ -0,0 +1,308 @@ +using CapriKit.IO; +using CapriKit.IO.Watchers; +using Microsoft.Extensions.Logging; +using System.Diagnostics; + +namespace CapriKit.AssetPipeline; + +/// +/// Rebuilds, reloads and hot-swaps tracked assets whenever one of the files they were built from changes. +/// Rebuilding and reloading happen on the thread pool, only the final hot-swap runs on the main thread +/// (see ). +/// A failed rebuild, reload or hot-swap only means that the asset keeps its current contents, it never +/// invalidates a live asset and never leaves a lease behind in the . +/// Threading: may be called from any thread, and +/// must be called from the main thread. +/// +internal sealed partial class HotReloadManagerV3 : IDisposable +{ + private readonly ILogger Logger; + private readonly AssetCache Cache; + private readonly ScopedFileSystem FileSystem; + private readonly IVirtualFileSystemWatcher Watcher; + private readonly FileSystemEventQueue FileChanges; + private readonly TimeSpan Debounce; + + // Guards the two collections that Track (any thread) and Update (main thread) share. + private readonly Lock Lock; + private readonly Dictionary Tracked; + private readonly Dictionary> Dependents; + + // Only touched by the main thread. + private readonly HashSet Stale; + private readonly Dictionary> InFlight; + private long lastChange; + + private bool isDisposed; + + /// + /// The optional debounce is how long to wait after the last relevant file change before rebuilding. + /// Editors write their buffer in several steps, without a pause we would rebuild the same asset once per + /// step and read half-written files. + /// + public HotReloadManagerV3(ILoggerFactory loggerFactory, AssetCache cache, ScopedFileSystem fileSystem, TimeSpan? debounce = null) + { + Logger = loggerFactory.CreateLogger(); + Cache = cache; + FileSystem = fileSystem; + Debounce = debounce ?? TimeSpan.FromSeconds(0.5); + + Lock = new(); + Tracked = []; + Dependents = []; + Stale = []; + InFlight = []; + + Watcher = FileSystem.Watch(); + FileChanges = new FileSystemEventQueue(Watcher); + } + + /// + /// Registers an asset so that it is rebuilt, reloaded and hot-swapped whenever one of the files it was + /// built from changes. Tracking the same asset more than once is a no-op. + /// Threading: thread-safe, may be called from any thread at any time. + /// + public void Track(Asset asset, IAssetTranscoder transcoder) + where TAsset : class + { + lock (Lock) + { + // After disposal we no longer listen for file changes, so tracking would only grow the maps. + if (isDisposed) { return; } + + // The asset manager materializes an asset once per outstanding handle, so the same asset arrives + // here several times. Everything we store comes from the build, so the first registration wins. + if (Tracked.ContainsKey(asset.Id)) { return; } + + var tracked = new TrackedAsset(asset, transcoder); + Tracked.Add(asset.Id, tracked); + RegisterDependencies(tracked); + } + } + + /// + /// Reacts to file changes, starts rebuilding the assets those files affect and hot-swaps the assets that + /// finished rebuilding. Every step is bounded work, the expensive rebuilding and reloading happens on the + /// thread pool so that the main thread only pays for the hot-swap itself. + /// Threading: must only be called from the main thread. + /// + public void Update() + { + if (isDisposed) { return; } + + MarkStaleAssets(); + + // Wait for the dust to settle so that a single save does not trigger a burst of rebuilds. + if (Stopwatch.GetElapsedTime(lastChange) >= Debounce) + { + StartReloads(); + } + + FinishReloads(); + } + + /// + /// Stops listening for file changes, abandons everything that has not started yet and finishes the + /// rebuilds that are already running so that their leases and freshly built data are handed back. + /// Threading: must only be called from the main thread. + /// + public void Dispose() + { + lock (Lock) + { + if (isDisposed) { return; } + isDisposed = true; + } + + Watcher.Stop(); + Stale.Clear(); + + // The running rebuilds hold a lease and own freshly built data that only the main thread can dispose + // of, so instead of abandoning them we wait and then finish them through the regular path. + try + { + Task.WaitAll([.. InFlight.Values]); + } + catch (AggregateException) + { + // Failures are reported and cleaned up per asset by FinishReloads + } + + FinishReloads(); + } + + /// + /// Marks every asset that depends on a changed file as stale. + /// + private void MarkStaleAssets() + { + lock (Lock) + { + while (FileChanges.TryDequeue(out var change)) + { + if (!Dependents.TryGetValue(change.File, out var dependents)) { continue; } + + // Only changes we actually care about restart the debounce window, otherwise unrelated + // writes (such as the asset pipeline writing its own build files) could postpone a rebuild. + lastChange = Stopwatch.GetTimestamp(); + + foreach (var id in dependents) + { + if (Stale.Add(id)) + { + LogAssetStale(Logger, change.File, id); + } + } + } + } + } + + /// + /// Starts rebuilding and reloading every stale asset on the thread pool. + /// + private void StartReloads() + { + if (Stale.Count == 0) { return; } + + lock (Lock) + { + foreach (var id in Stale.ToArray()) + { + // Let the running rebuild finish first. The asset stays stale so we pick it up again + // afterwards, which is exactly what we want because its files changed once more. + if (InFlight.ContainsKey(id)) { continue; } + + Stale.Remove(id); + + if (!Tracked.TryGetValue(id, out var tracked)) { continue; } + + if (tracked.TryStartReload(Cache, FileSystem, out var reload)) + { + InFlight.Add(id, reload); + LogReloadStarted(Logger, id); + } + else + { + // The cache is the authority on liveness: no entry means nobody uses this asset anymore. + UntrackAsset(tracked); + LogUntracked(Logger, id); + } + } + } + } + + /// + /// Hot-swaps every asset that finished rebuilding and returns the lease that its rebuild took. + /// + private void FinishReloads() + { + if (InFlight.Count == 0) { return; } + + foreach (var (id, reload) in InFlight.ToArray()) + { + if (!reload.IsCompleted) { continue; } + + InFlight.Remove(id); + FinishReload(id, reload); + } + } + + private void FinishReload(AssetId id, Task reload) + { + try + { + if (!reload.IsCompletedSuccessfully) + { + // Nothing was touched yet, so the asset simply keeps the contents it already had. + LogReloadFailed(Logger, id, reload.Exception!); + return; + } + + var reloaded = reload.Result; + + // Deliberately outside of the lock: the transcoder runs code we do not control here. + reloaded.HotSwap(); + + UpdateDependencies(id, reloaded.Dependencies); + LogHotSwapped(Logger, id); + } + catch (Exception ex) + { + // A transcoder that fails half-way leaves the asset in whatever state it made of it, all we can + // do is report it. The alternative, throwing, would take down the game over a development feature. + LogHotSwapFailed(Logger, id, ex); + } + finally + { + // Balances the lease that TryStartReload took, whether we managed to hot-swap or not. + Cache.Return(id); + } + } + + /// + /// Replaces the dependencies of an asset with the ones its latest build read. + /// + private void UpdateDependencies(AssetId id, IReadOnlyList dependencies) + { + lock (Lock) + { + if (Tracked.TryGetValue(id, out var tracked)) + { + UnregisterDependencies(tracked); + tracked.Dependencies = dependencies; + RegisterDependencies(tracked); + } + } + } + + // The three methods below must be called while holding the lock. + + private void RegisterDependencies(TrackedAsset tracked) + { + foreach (var (file, _) in tracked.Dependencies) + { + if (!Dependents.TryGetValue(file, out var ids)) + { + ids = []; + Dependents.Add(file, ids); + } + + ids.Add(tracked.Id); + } + } + + private void UnregisterDependencies(TrackedAsset tracked) + { + foreach (var (file, _) in tracked.Dependencies) + { + if (Dependents.TryGetValue(file, out var ids) && ids.Remove(tracked.Id) && ids.Count == 0) + { + Dependents.Remove(file); + } + } + } + + private void UntrackAsset(TrackedAsset tracked) + { + UnregisterDependencies(tracked); + Tracked.Remove(tracked.Id); + } + + [LoggerMessage(Level = LogLevel.Information, Message = "Detected change in file: {file}, marking asset: {asset} as stale")] + private static partial void LogAssetStale(ILogger logger, FilePath file, AssetId asset); + + [LoggerMessage(Level = LogLevel.Information, Message = "Started rebuilding and reloading asset: {asset}")] + private static partial void LogReloadStarted(ILogger logger, AssetId asset); + + [LoggerMessage(Level = LogLevel.Error, Message = "Rebuilding or reloading asset: {asset} failed, it keeps its current contents")] + private static partial void LogReloadFailed(ILogger logger, AssetId asset, Exception exception); + + [LoggerMessage(Level = LogLevel.Information, Message = "Hot-swapped asset: {asset}")] + private static partial void LogHotSwapped(ILogger logger, AssetId asset); + + [LoggerMessage(Level = LogLevel.Error, Message = "Hot-swapping asset: {asset} failed")] + private static partial void LogHotSwapFailed(ILogger logger, AssetId asset, Exception exception); + + [LoggerMessage(Level = LogLevel.Information, Message = "Stopped tracking asset: {asset}, it is no longer in the cache")] + private static partial void LogUntracked(ILogger logger, AssetId asset); +} diff --git a/source/CapriKit.AssetPipeline/HotReloadPipeline.cs b/source/CapriKit.AssetPipeline/HotReloadPipeline.cs new file mode 100644 index 0000000..7b5122d --- /dev/null +++ b/source/CapriKit.AssetPipeline/HotReloadPipeline.cs @@ -0,0 +1,77 @@ +//using CapriKit.Concurrency.Primitives; + +//namespace CapriKit.AssetPipeline; + +//internal abstract record HotSwapRecipe +//{ +// public abstract void HotSwap(); +//} + + +//internal abstract record ReloadRecipe +//{ +// public abstract Task Reload(); +//} + +//internal abstract record ReloadRecipe(AssetId Id, AssetBuildMetaData Metadata, IAssetTranscoder Transcoder) +// : ReloadRecipe +// where TAsset : class +//{ +// public override Task Reload() +// { + +// } +//} + + +//internal sealed class HotReloadPipeline +//{ +// private readonly LightweightChannel WaitingForRebuild = new(); +// private readonly LightweightChannel WaitingForHotSwap = new(); + +// private volatile bool isEnabled = true; +// private volatile bool isWorking = false; +// private Task? reloadTask = null; +// private Lock StateLock = new Lock(); + +// public bool TryEnter(ReloadRecipe recipe) +// { +// lock (StateLock) +// { +// if (!isEnabled) { return false; } + +// WaitingForRebuild.Write(recipe); +// return true; +// } +// } + +// public void Update() +// { +// lock (StateLock) +// { +// if (isWorking) +// { +// if (reloadTask != null && reloadTask.IsCompletedSuccessfully) +// { +// WaitingForHotSwap.Write(reloadTask.Result); +// } +// } +// else +// { + +// if (WaitingForRebuild.TryRead(out var rebuild)) +// { +// reloadTask = rebuild.Reload(); +// reloadTask.Start(); // TODO: is this necessary? +// isWorking = true; +// } +// } +// } + + +// while (WaitingForHotSwap.TryRead(out var hotswap)) +// { +// hotswap.HotSwap(); +// } +// } +//} diff --git a/source/CapriKit.AssetPipeline/TrackedAsset.cs b/source/CapriKit.AssetPipeline/TrackedAsset.cs new file mode 100644 index 0000000..0c1d6da --- /dev/null +++ b/source/CapriKit.AssetPipeline/TrackedAsset.cs @@ -0,0 +1,79 @@ +using CapriKit.IO; +using System.Diagnostics.CodeAnalysis; + +namespace CapriKit.AssetPipeline; + +/// +/// A freshly rebuilt asset that is waiting for the main thread to move it into the live instance. +/// +/// Files read while rebuilding, used to keep the file-to-asset map up-to-date. +/// +/// Moves the rebuilt data into the live instance. Threading: main thread only, see . +/// +internal sealed record ReloadedAsset(IReadOnlyList Dependencies, Action HotSwap); + +/// +/// Everything the needs to rebuild a single asset. The asset and settings +/// types are erased so that assets of every type can live in one collection. +/// +internal abstract class TrackedAsset(AssetId id, IReadOnlyList dependencies) +{ + public AssetId Id { get; } = id; + + /// + /// The files that the most recent successful build of this asset read. + /// Threading: only touched by the main thread while it holds the manager's lock. + /// + public IReadOnlyList Dependencies { get; set; } = dependencies; + + /// + /// Leases the live asset from the cache and then starts rebuilding it on the thread pool. Returns false if + /// the asset is no longer in the cache, in which case no lease was taken and nothing was started. + /// The lease is taken before the task starts so that the caller always knows whether a lease exists, the + /// caller owns that lease and must return it once has completed. + /// Threading: main thread only. + /// + public abstract bool TryStartReload(AssetCache cache, IVirtualFileSystem fileSystem, [NotNullWhen(true)] out Task? reload); +} + +/// +internal sealed class TrackedAsset : TrackedAsset + where TAsset : class +{ + private readonly TSettings Settings; + private readonly IAssetTranscoder Transcoder; + + public TrackedAsset(Asset asset, IAssetTranscoder transcoder) + : base(asset.Id, asset.BuildMetaData.Dependencies) + { + Settings = asset.BuildMetaData.Settings; + Transcoder = transcoder; + } + + public override bool TryStartReload(AssetCache cache, IVirtualFileSystem fileSystem, [NotNullWhen(true)] out Task? reload) + { + // The lease pins the live instance for the entire rebuild, so the background thread never has to + // wonder whether the object it is going to hot-swap into still exists. + if (!cache.TryLease(Id, out var live)) + { + reload = null; + return false; + } + + reload = Task.Run(() => Reload(live, fileSystem)); + return true; + } + + private async Task Reload(TAsset live, IVirtualFileSystem fileSystem) + { + // Build into memory rather than over the existing build on disk: other threads may be reading that + // file to load the very same asset and overwriting it underneath them would fail those loads. + using var stream = new MemoryStream(); + await AssetEncoder.Encode(Id, Transcoder, Settings, fileSystem, stream); + + stream.Seek(0, SeekOrigin.Begin); + var rebuilt = await AssetDecoder.Decode(Id, Transcoder, fileSystem, stream); + + return new ReloadedAsset(rebuilt.BuildMetaData.Dependencies, () => Transcoder.HotSwap(live, rebuilt.Value)); + } +} diff --git a/source/CapriKit.Tests/AssetPipeline/HotReloadManagerV3Tests.cs b/source/CapriKit.Tests/AssetPipeline/HotReloadManagerV3Tests.cs new file mode 100644 index 0000000..7318daa --- /dev/null +++ b/source/CapriKit.Tests/AssetPipeline/HotReloadManagerV3Tests.cs @@ -0,0 +1,123 @@ +using CapriKit.AssetPipeline; +using CapriKit.IO; +using CapriKit.IO.Streams; +using Microsoft.Extensions.Logging.Abstractions; +using System.Buffers; + +namespace CapriKit.Tests.AssetPipeline; + +internal class HotReloadManagerV3Tests +{ + private static readonly FilePath AssetFile = new("Hello.txt"); + private static readonly TimeSpan NoDebounce = TimeSpan.Zero; + + [Test] + public async Task Update() + { + // Arrange: build, load and cache an asset the way the asset manager would + var fileSystem = new InMemoryFileSystem().ScopedTo("C:/Test"); + await fileSystem.WriteAllText(AssetFile, "Hello World"); + + var transcoder = new TextTranscoder(); + var id = new AssetId(AssetFile); + + await AssetEncoder.Encode(id, transcoder, default, fileSystem); + var asset = await AssetDecoder.Decode(id, transcoder, fileSystem); + + using var cache = new AssetCache(); + var live = cache.PutOrLease(id, asset.Value); + + using var sut = new HotReloadManagerV3(NullLoggerFactory.Instance, cache, fileSystem, NoDebounce); + sut.Track(asset, transcoder); + + // Act: change the file the asset was built from + await fileSystem.WriteAllText(AssetFile, "Goodbye World"); + + await Assert.That(() => + { + sut.Update(); + return live.Text; + }) + .Eventually(v => v.IsEqualTo("Goodbye World"), TimeSpan.FromSeconds(5)); + + // Assert: the caller's instance was updated in place, and we left no lease behind + await Assert.That(live.Text).IsEqualTo("Goodbye World"); + cache.Return(id); + } + + [Test] + public async Task Update_RebuildFails() + { + // Arrange: build, load and cache an asset, then make every following rebuild fail + var fileSystem = new InMemoryFileSystem().ScopedTo("C:/Test"); + await fileSystem.WriteAllText(AssetFile, "Hello World"); + + var transcoder = new FailingTranscoder(); + var id = new AssetId(AssetFile); + + await AssetEncoder.Encode(id, transcoder, default, fileSystem); + var asset = await AssetDecoder.Decode(id, transcoder, fileSystem); + + using var cache = new AssetCache(); + var live = cache.PutOrLease(id, asset.Value); + + var sut = new HotReloadManagerV3(NullLoggerFactory.Instance, cache, fileSystem, NoDebounce); + sut.Track(asset, transcoder); + transcoder.ShouldFail = true; + + // Act: one update starts the rebuild, disposing waits for it and finishes it + await fileSystem.WriteAllText(AssetFile, "Goodbye World"); + sut.Update(); + sut.Dispose(); + + // Assert: the rebuild really was attempted, the live asset kept its contents, and returning the last + // lease empties the cache. If the manager leaked its lease the cache throws when it is disposed. + await Assert.That(transcoder.FailedAttempts).IsEqualTo(1); + await Assert.That(live.Text).IsEqualTo("Hello World"); + cache.Return(id); + } + + // Hot-swapping needs an asset that can be updated in place, so a mutable holder instead of a plain string + private sealed class TextAsset(string text) + { + public string Text { get; set; } = text; + } + + private class TextTranscoder() : NoSettingsTranscoder(Guid.Parse("{6E4A1D0C-1F73-4C4E-9D2E-0B7F5C6A9E31}"), 1) + { + public override TextAsset Decode(AssetId id, ref SequenceReader reader) + { + return new TextAsset(reader.ReadString()); + } + + public override async Task Encode(AssetId id, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer) + { + var text = await fileSystem.ReadAllText(id.Path); + writer.Write(text); + } + + public override void HotSwap(TextAsset instance, TextAsset newParts) + { + instance.Text = newParts.Text; + } + } + + private sealed class FailingTranscoder : TextTranscoder + { + public bool ShouldFail { get; set; } + + /// Written on a thread pool thread, only safe to read once the rebuild completed. + public int FailedAttempts { get; private set; } + + public override Task Encode(AssetId id, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer) + { + if (ShouldFail) + { + FailedAttempts++; + throw new InvalidOperationException("Rebuilding this asset fails on purpose"); + } + + return base.Encode(id, fileSystem, writer); + } + } +} From f41d279ac5dd64316a68f190e045c70dd84b5594 Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Tue, 18 Aug 2026 20:49:28 +0200 Subject: [PATCH 44/53] Finalize the asset pipeline, only a few open questions --- Research/AssetPipeline.md | 27 - Research/AssetPipelineArchitecture.md | 234 ------ ...pelineHotReloadWithoutAssetReferences.html | 692 ----------------- Research/AssetPipelineLoadingGroups.md | 298 ------- Research/AssetPipelineLoadingGroupsV2.md | 273 ------- Research/AssetPipelineReview.md | 292 ------- Research/AssetPipelineRewriteReview.md | 726 ------------------ Research/AssetPipelineVNextReview.html | 675 ---------------- .../AssetPipelineVNextReviewContinued.html | 696 ----------------- Research/HowIDidIt.md | 164 ---- ...nseToFindingsOfAssetPipelineVNextReview.md | 29 - source/CapriKit.AssetPipeline/Asset.cs | 9 + .../{AssetBundle.cs => AssetBundleLoader.cs} | 26 +- source/CapriKit.AssetPipeline/AssetHandle.cs | 11 +- source/CapriKit.AssetPipeline/AssetManager.cs | 23 +- .../{AssetCache.cs => AssetPool.cs} | 11 +- .../HotReloadManager.cs | 357 +++++---- .../HotReloadManagerV2.cs | 230 ------ .../HotReloadManagerV3.cs | 308 -------- .../HotReloadPipeline.cs | 77 -- .../CapriKit.AssetPipeline/HotReloadable.cs | 47 -- .../IAssetTranscoder.cs | 29 +- source/CapriKit.AssetPipeline/README.md | 51 ++ source/CapriKit.AssetPipeline/TODO.md | 7 - source/CapriKit.AssetPipeline/TrackedAsset.cs | 6 +- .../AssetPipeline/AssetManagerTests.cs | 49 +- .../AssetPipeline/HotReloadManagerTests.cs | 88 ++- .../AssetPipeline/HotReloadManagerV3Tests.cs | 123 --- 28 files changed, 441 insertions(+), 5117 deletions(-) delete mode 100644 Research/AssetPipeline.md delete mode 100644 Research/AssetPipelineArchitecture.md delete mode 100644 Research/AssetPipelineHotReloadWithoutAssetReferences.html delete mode 100644 Research/AssetPipelineLoadingGroups.md delete mode 100644 Research/AssetPipelineLoadingGroupsV2.md delete mode 100644 Research/AssetPipelineReview.md delete mode 100644 Research/AssetPipelineRewriteReview.md delete mode 100644 Research/AssetPipelineVNextReview.html delete mode 100644 Research/AssetPipelineVNextReviewContinued.html delete mode 100644 Research/HowIDidIt.md delete mode 100644 Research/ResponseToFindingsOfAssetPipelineVNextReview.md rename source/CapriKit.AssetPipeline/{AssetBundle.cs => AssetBundleLoader.cs} (64%) rename source/CapriKit.AssetPipeline/{AssetCache.cs => AssetPool.cs} (89%) delete mode 100644 source/CapriKit.AssetPipeline/HotReloadManagerV2.cs delete mode 100644 source/CapriKit.AssetPipeline/HotReloadManagerV3.cs delete mode 100644 source/CapriKit.AssetPipeline/HotReloadPipeline.cs delete mode 100644 source/CapriKit.AssetPipeline/HotReloadable.cs create mode 100644 source/CapriKit.AssetPipeline/README.md delete mode 100644 source/CapriKit.AssetPipeline/TODO.md delete mode 100644 source/CapriKit.Tests/AssetPipeline/HotReloadManagerV3Tests.cs diff --git a/Research/AssetPipeline.md b/Research/AssetPipeline.md deleted file mode 100644 index 5345d53..0000000 --- a/Research/AssetPipeline.md +++ /dev/null @@ -1,27 +0,0 @@ -# Asset Pipeline -Now that I have support for encoding textures offline and then loading/transcoding them when the game runs (see CapriKit.SuperCompressed) it is time to start work on a real asset pipeline. I want you to help me create a high-level architecture for it. The expected output (when we are ready for that) is a markdown file in the `Research` folder. - -# Goals -1. Encoding/compile/compress assets into formats that makes them easy to ship and easy load -2. Load the assets when requested by the game into ready-to-use objects (for example, when done a texture is uploaded to the GPU and ready to be referenced) - -## Subgoals -1. Flexible enough to support textures, shaders, models and audio in a similar process -2. Support for detecting changes on-disk and then rebuilding and hot-reloading them -3. Loading a groups of assets (bundles) in parallel while the main loop keeps running uninterrupted. - 3a. I am thinking of a process similar to the parallel loading of test screens in the background in `C:\projects\csharp\CapriKit\source\CapriKit.Tests.Tool\Program.cs` with a `LightweightChannel` `C:\projects\csharp\CapriKit\source\CapriKit.Concurrency\Primitives\LightweightChannel.cs` - 3b. I wonder if I need two steps here, full background work and then coordination with the main loop to finish the loading of things like textures that need to be uploaded to GPU memory, which I think you cannot do in parallel/without a DeviceContext and help of the main thread. - 3c. I need a way to identify assets (maybe just a record `Asset` or `Asset` which holds the relative path to the asset). If I load a bundle I should be able to wait for the bundle to be fully loaded and then get the concrete Texture, Model, etc... object from the bundle using that record. - 3d. Somehow that Texture, model, etc.. object needs to have a point of indirection to support that hot reloading - - -I am saying bundle here as a helpful abstraction in code, but on disk I want each file to stay independent. I know it is a common optimization to compress multiple assets into one file and load those together (better disk IO) but I think that will complicate hot reloading and other IO code. (Happy to be proven wrong though, so maybe discuss it as an extra option but assume for now we are not doing it). - -## Assumptions -1. Assume assets do not need to carry extra information for the content pipeline. For example, if a texture should be loaded as normal map, srgb or linear texture is determined by the caller at run-time, not via an extra config file. -2. Assume that if an asset is loaded it stays loaded, we're more going for games like Factorio, Stationeers, Satisfactory and not for level-based games. - -## Examples -I have created a similar system before. See `C:\projects\csharp\MiniEngine3\src\Mini.Engine.Content\` and especially `C:\projects\csharp\MiniEngine3\src\Mini.Engine.Content\ContentManager.cs` and `C:\projects\csharp\MiniEngine3\src\Mini.Engine.Content\ContentProcessor.cs` but there were a couple of problems -- Background loading was super hacky -- A lot of boilerplate code, for example in the folder `C:\projects\csharp\MiniEngine3\src\Mini.Engine.Content\Shaders\` you can see that I needed 3 classes for each type of shader. I really want to have to use a lot less code. diff --git a/Research/AssetPipelineArchitecture.md b/Research/AssetPipelineArchitecture.md deleted file mode 100644 index fd0ff6d..0000000 --- a/Research/AssetPipelineArchitecture.md +++ /dev/null @@ -1,234 +0,0 @@ -# Asset Pipeline — High-Level Architecture - -Companion to `AssetPipeline.md` (the brief). Status: draft for discussion, 2026-07-10. - -## Decisions - -| Decision | Choice | Rationale | -|---|---|---| -| Build timing | **Hybrid** | Dev builds compile stale assets on demand into a cache; a CLI step produces the same output for shipping. Shipped games load precompiled assets only and carry no compiler code. | -| Asset relationships | **Flat** | No cross-asset references. Game code composes (loads a model's geometry and its textures separately). Kills the dependency-graph machinery that made MiniEngine3 complex. | -| GPU coupling | **Direct DX11 dependency** | The pipeline references `CapriKit.DirectX11` and produces GPU resources itself. Fewer abstractions to understand and maintain. | -| Shaders | **Offline bytecode** | HLSL → DXBC at compile time. `CapriKit.Generators.HLSL` keeps generating the C#-side metadata (structs, input element descriptions, entry points) unchanged; the pipeline only takes over bytecode compilation. | - -## Big Picture - -``` - source tree compiled tree runtime - (loose files) (loose files, mirrored) - ┌──────────────┐ compile ┌──────────────┐ load ┌───────────────┐ - │ grass.png │ ─────────► │ grass.cka │ ─────────► │ Texture2D │ - │ basic.hlsl │ IAsset- │ basic.cka │ IAsset- │ VertexShader..│ - │ tree.gltf │ Compiler │ tree.cka │ Loader │ Model │ - │ click.wav │ │ click.cka │ │ AudioClip │ - └──────────────┘ └──────────────┘ └───────────────┘ - dev machine only dev cache == ship content game process -``` - -Two phases, two families of pluggable pieces: - -- **Compilers** (dev machine / build server): source format → compiled container. Slow, thorough, offline. -- **Loaders** (game process): compiled container → ready-to-use object (texture on the GPU, playable audio clip). Fast, allocation-conscious. - -An asset is identified by its **source-relative path**, and every source file maps 1:1 to -one compiled file with the same relative path (different extension). This 1:1 rule is what -keeps identity, staleness checks, and file watching trivial. - -## Phase 1: Compiling - -```csharp -public interface IAssetCompiler -{ - IReadOnlySet SupportedExtensions { get; } - int Version { get; } - - // Reads source via a *tracking* file system so every file touched - // (e.g. #included HLSL) is recorded as an input of this asset. - void Compile(AssetId id, ITrackingReadOnlyFileSystem source, Stream output); -} -``` - -### Compiled container (`.cka` — "CapriKit Asset") - -One small binary envelope shared by all types, so staleness and loading logic is written once: - -- Header: magic, container version, compiler type + version, hashes of all input files. -- Body: one or more named blobs (a KTX2 payload; a DXBC blob per shader entry point; vertex + index buffers). - -### Staleness & the hybrid model - -An asset is stale when the compiled file is missing, or the stored input hashes / compiler -version no longer match. The check is identical everywhere; only *who runs it* differs: - -- **Dev**: the game process checks staleness when an asset is first requested and recompiles - inline (on the worker thread that is loading it) before loading. First run is slow, then it's cache hits. -- **Ship**: the CLI tool walks the source tree, compiles everything stale, and the resulting - compiled tree *is* the shipped content folder. Shipped builds skip the staleness check entirely - and never reference the compiler assemblies. - -### Per-type mapping - -| Type | Source | Compile step | Compiled body | -|---|---|---|---| -| Texture | png, jpg, ... | `CapriKit.SuperCompressed` encoder (mips baked in) | KTX2 | -| Shader | hlsl | DXC/FXC per `#pragma`-marked entry point | DXBC blob per entry point | -| Model | tbd (gltf?) | parse, triangulate, build interleaved buffers | vertex + index blobs | -| Audio | wav, ogg | tbd (likely near pass-through) | PCM or compressed blob | - -Note that input-file tracking (an `.hlsl` including `common.hlsli`) is a *build* dependency, -not a runtime asset reference — it exists only so staleness and hot reload know that touching -`common.hlsli` dirties every shader that included it. The "flat assets" decision is untouched. - -## Phase 2: Loading - -### Identity and typed access - -```csharp -public sealed record Asset(string Path); // relative path, forward slashes, lower case - -public static class GameAssets // game code declares what it uses -{ - public static readonly Asset Grass = new("textures/grass.png"); - // snip -} -``` - -Per assumption 1, interpretation (sRGB vs linear, transcode target, ...) is supplied by the -caller **at request time** as an optional per-type options record; nothing is persisted. - -### Bundles and background loading - -This generalizes the pattern already proven in `CapriKit.Tests.Tool/Program.cs`: -`BackgroundWorker` jobs write results (or one exception) into a `LightweightChannel`, -and the main loop drains it once per frame. - -```csharp -var bundle = assets.StartLoading([GameAssets.Grass, GameAssets.Tree /* snip */]); - -// in the main loop, once per frame: -assets.Update(); // drains channels, publishes finished assets, applies hot reloads - -if (bundle.IsLoaded) // all jobs finished (faulted bundles rethrow here) -{ - AssetRef grass = bundle.Get(GameAssets.Grass); -} -``` - -`bundle.Progress` (loaded/total) falls out for free for loading screens. - -### Threading: answering "do I need two steps?" (brief 3b) - -Mostly no, on D3D11. `ID3D11Device` is **free-threaded**: creating textures, buffers, and -shaders — including uploading initial data — is legal from any thread. Only the -**immediate `DeviceContext`** is single-threaded. Since mips are baked offline, nothing in -the current asset set needs the context at load time, so a worker thread can hand back a -fully GPU-resident texture. - -The main-thread coordination point still exists, but it is just `assets.Update()` draining -the channel: finished objects become *visible* to game code only on the main thread, which is -also the only place the registry is ever mutated — no locks anywhere. If a future asset type -does need context work (e.g. runtime `GenerateMips`), it can queue an action that `Update()` -executes; that seam costs nothing today. - -### Indirection for hot reload (3d) - -The registry maps `Asset` → `AssetRef`, created once and permanent (assumption 2: -loaded stays loaded, so no lifetime/refcount management at all). - -```csharp -public sealed class AssetRef -{ - public T Value { get; } // a field read, no lookup cost per use - public int Version { get; } // bumped on hot reload - // snip: internal setter used by the registry during Update() -} -``` - -Game code holds the `AssetRef` and reads `Value` when used. Derived objects (an input -layout built from a vertex shader blob) either poll `Version` or subscribe to a sparse -`Changed` event — needed rarely, mostly for shaders. - -## Hot Reload (dev only) - -``` -FileSystemWatcher (source tree) - → debounce (editors save in bursts) - → look up which assets have that file as an input (from stored input lists) - → recompile + reload on a worker, same code path as a normal load - → assets.Update() swaps AssetRef.Value on the main thread, bumps Version - → old object disposed (D3D11 keeps GPU resources alive while in flight, so this is safe) -``` - -Failures (syntax error in a shader) write the exception to the channel; `Update()` reports -it and keeps the old value live — a typo never kills the running game. - -## Project Layout - -| Project | Contents | Referenced by | -|---|---|---| -| `CapriKit.Assets` | `Asset`, registry, bundles, loaders, hot-reload swap. Refs `DirectX11`, `SuperCompressed` (transcode), `Concurrency`, `IO`. | game, always | -| `CapriKit.Assets.Compilers` | `IAssetCompiler` implementations, staleness, watcher. Refs `SuperCompressed` (encode), shader compiler. | game in dev builds; CLI tool | -| `CapriKit.Assets.Tool` | thin CLI over Compilers for the shipping build step | build scripts | - -Boilerplate budget per new asset type: **one compiler class + one loader class** -(MiniEngine3 needed processor + content wrapper + settings + serialization per type; the -shared container format and the no-settings/no-lifetime assumptions eliminate the rest). - -## Deliberately Out of Scope - -- Unloading / reference counting (assumption 2). -- Cross-asset references and cascading reloads (flat-assets decision). -- Per-asset metadata files (assumption 1). -- Packed archives — see appendix. - -## Risks & Open Points - -1. **Encoder settings vs assumption 1.** Normal maps and albedo textures ideally encode - differently (UASTC vs ETC1S, linear vs perceptual metrics). If one default proves - insufficient, a filename convention (`*_n.png`) is the escape hatch that doesn't violate - "no metadata files". -2. **Model and audio source formats** are undecided; the architecture only assumes "some - compiler produces blobs". Worth a separate research note before implementing those compilers. -3. **Input tracking in `CapriKit.IO`.** Compilers need a tracking read-only file system - (MiniEngine3's `TrackingVirtualFileSystem` is the precedent); check what `CapriKit.IO` - is missing. -4. **Native binary size.** The SuperCompressed native DLL contains encoder *and* transcoder; - shipped games carry encoder code they never call. Acceptable for now, splittable later. - -## Appendix: Packed Archives (not now, maybe later) - -Skipped because loose files make hot reload, staleness, and debugging trivially simple, and -the classic motivations (seek latency, file-handle overhead) matter little on modern SSDs. -Remaining real benefits: fewer/smaller patch artifacts, whole-archive compression, and light -obfuscation. - -The door stays open cheaply: loaders read compiled containers through -`IReadOnlyVirtualFileSystem`, so a future `PackedFileSystem` implementation (one archive = -one mounted file system) would slot in without touching any pipeline code. If it ever -happens, pack as a *post-step* over the compiled tree so the compile pipeline never knows. - -## Remark: uploading textures from a worker thread - -`UpdateSubresource`/`Map` need the immediate context (main thread), but they are only for -writing into a resource that *already exists* (`Default`/`Dynamic` usage). Asset textures -instead pass their pixels as initial data to `ID3D11Device::CreateTexture2D`, which is -free-threaded: one `SubresourceData` entry per subresource (mip × array slice), copied by -the driver *during* the create call. Use `ResourceUsage.Immutable` — it requires initial -data at creation, forbids later updates, and lets the driver optimize. - -```csharp -var subresources = new SubresourceData[mipCount]; -for (var mip = 0; mip < mipCount; mip++) -{ - // for BC formats: rowPitch = Math.Max(1, (mipWidth + 3) / 4) * bytesPerBlock - subresources[mip] = new SubresourceData(pointerToMipData, rowPitch); -} -var desc = new Texture2DDescription(format, width, height, mipLevels: mipCount, /* snip */ usage: ResourceUsage.Immutable); -var texture = device.CreateTexture2D(desc, subresources); -// pointers only need to stay pinned until CreateTexture2D returns — the copy is synchronous -``` - -Caveats: free-threaded means thread-safe, not necessarily concurrent — without -`DriverConcurrentCreates` (see `CheckFeatureSupport(D3D11_FEATURE_THREADING)`) creates -serialize on a device-wide lock, which is still correct. And none of this holds if the -device were created with `D3D11_CREATE_DEVICE_SINGLETHREADED` (ours is not, see `Device.cs`). diff --git a/Research/AssetPipelineHotReloadWithoutAssetReferences.html b/Research/AssetPipelineHotReloadWithoutAssetReferences.html deleted file mode 100644 index d00079a..0000000 --- a/Research/AssetPipelineHotReloadWithoutAssetReferences.html +++ /dev/null @@ -1,692 +0,0 @@ - - - - - -Hot Reload Without Asset References - - - - -
-
-
CapriKit · feature/asset_pipeline · design exploration
-

Hot Reload Without Asset References

-

What happens to the code, and to the number of things you have to keep in your head, if - HotReloadManager stops holding assets entirely — and instead asks AssetCache for a - lease whenever a file changes.

-
- Baseline 321 lines — HotReloadManager.cs + HotReloadable.cs - Status exploration, nothing implemented - Date 2026-08-15 -
- -
-
Concepts removed
7
two added
-
Findings closed
H3 H5
by construction, not by fix
-
Lines
−27
321 → ~294, only −8%
-
New locks
0
all cache calls main-thread
-
Gets harder
M5
shutdown with a live lease
-
Verdict
Do it
certainty, not size
-
-
-
- -
-
-

The idea, and why it works

- -

The contract changes from a sentence with a hedge in it to a sentence without one.

- -
-
-

Today

-

“HotReloadManager keeps weak references to the assets it has - seen, and when a file changes it picks the first one the GC has not collected yet and hopes that is the one the - engine is actually using.”

-
-
-

Proposed

-

“HotReloadManager knows how to rebuild an asset. - AssetCache knows whether anyone still wants it. When a file changes, the manager asks the - cache for a lease; if it gets one, it rebuilds.”

-
-
- -

The whole design rests on one property that is already true in the cache you have written, and it is worth being - precise about it because everything else follows:

- - 
// AssetCache.Return, line 93-99
-entry.RefCount--;
-// Evict items immediately, but only dispose of them in Collect
-if (entry.RefCount <= 0)
-{
-    Entries.Remove(id);          // <-- gone from Entries the instant the last user lets go
-    PendingDispose.Enqueue(entry);
-}
- -

Eviction from Entries is immediate and deterministic; only the Dispose call - is deferred to Collect(). So TryLease returning false means exactly - “no live user holds this asset” — not “the GC happens to have got round to it”. That is - the question HotReloadable.IsAlive has been trying and failing to answer since it was written, and the - cache has been answering it correctly all along. Nobody was asking.

- -
- The two guarantees a lease buys you -

1. Liveness. TryLease succeeds ⇒ at least one real user holds this asset. Entries - at refcount 0 are not in Entries, so they cannot be leased. H3 — hot-swapping an asset - Collect already disposed — becomes unrepresentable rather than merely unlikely.

-

2. Identity stability. While you hold the lease the entry cannot be evicted, and - PutOrLease only ever inserts when an entry is absent. So the instance you leased is still the - instance the cache holds when your rebuild lands one frame later. That is a proof, not a hope — and it deletes - the whole stale-swap detection layer (IsCurrent, HotSwapAction.Target, - DiscardNewParts) that the previous report proposed for exactly this window.

-
-
-
- -
-
-

What the manager still has to hold

- -

This is where the idea meets the type system, and it is the one place where the result is less tidy than - the sentence above suggests.

- -

You cannot delete the per-asset object. On a file-change event the manager has an AssetId and nothing - else — no TAsset, no TSettings. But Cache.TryLease<TAsset> is - generic, and AssetEncoder.Encode needs both type parameters. Something has to carry those types from - registration time to reload time, and in C# that something is an object with an abstract base:

- - 
internal abstract class ReloadRecipe(AssetId id)
-{
-    public AssetId Id { get; } = id;
-    public IReadOnlyList<Dependency> Dependencies { get; set; } = [];
-
-    /// <summary>
-    /// Leases the live instance from the cache and starts a rebuild on the thread pool.
-    /// Returns false when no live instance exists, which means this recipe is dead and can be forgotten.
-    /// The lease is released in HotSwapPending, exactly once, whether the rebuild succeeds or fails.
-    /// </summary>
-    public abstract bool TryStartReload(AssetCache cache, IVirtualFileSystem fileSystem,
-        ConcurrentQueue<ReloadResult> results, out Task reload);
-}
- -

So the class hierarchy survives. What changes is what it holds, and that is the entire point:

- -
-
-

HotReloadable<TAsset, TSettings> — today

- 
private readonly WeakReference<TAsset> Instance;   // <-- an asset
-private readonly TSettings Settings;
-private readonly IAssetTranscoder<TAsset, TSettings> Transcoder;
-
-public override bool IsAlive => Instance.TryGetTarget(out var _);
-
-
-

ReloadRecipe<TAsset, TSettings> — proposed

- 
private readonly TSettings Settings;
-private readonly IAssetTranscoder<TAsset, TSettings> Transcoder;
-
-// no asset field, so no IsAlive, and nothing to keep alive
-
-
- -

One field goes. Everything that field's ambiguity forced goes with it, and that is a much longer list than - one field: IsAlive on the base and the override, the candidate-selection loop and its five-line TODO in - ReloadOne, the List<HotReloadable> that only existed because the manager could not tell - the instances apart, the broken dedupe guard (H5), the pruned list that gets built and discarded, and - UnTrack, which has no caller today and now never needs one.

- -
- Make it structural, not documentary -

“Holds no asset references” should be visible in the signature rather than true by inspection. Today - Track takes the whole Asset<TAsset, TSettings> and simply chooses not to keep - .Value. Narrow the parameters instead, and the property becomes something the compiler enforces:

- 
// before: the asset is right there, one careless line away from being captured
-public void Track<TAsset, TSettings>(Asset<TAsset, TSettings> asset, IAssetTranscoder<TAsset, TSettings> transcoder)
-
-// after: there is nothing to capture
-public void Track<TAsset, TSettings>(AssetId id, TSettings settings,
-    IReadOnlyList<Dependency> dependencies, IAssetTranscoder<TAsset, TSettings> transcoder)
-

Same reasoning as AssetHandle having no .Value accessor: the invariant that survives is - the one the type system carries.

-
-
-
- -
-
-

The code, before and after

- -

Track — 22 lines to 6

-
-
-

HotReloadManager.cs:58-79

- 
lock (TrackingLock)
-{
-    var reloadable = new HotReloadable<TAsset, TSettings>(asset, transcoder);
-    if (Tracked.TryGetValue(asset.Id, out var assets))
-    {
-        // TODO: this is broken!!!!
-        // Prevent adding the exact same instance multiple times
-        if (assets.Any(a => object.ReferenceEquals(a, asset))) { return; }
-        assets.Add(reloadable);
-    }
-    else
-    {
-        Tracked[asset.Id] = [reloadable];
-    }
-
-    RegisterFileDependencies(asset.Id, asset.BuildMetaData.Dependencies);
-}
-
-
-

proposed

- 
// One live instance per id is the cache's guarantee,
-// so one recipe per id is all we can ever need.
-Tracked[id] = new ReloadRecipe<TAsset, TSettings>(id, settings, transcoder)
-{
-    Dependencies = dependencies
-};
-
-ReplaceFileDependencies(id, dependencies);
-
-
-

The dedupe question that H5 got wrong does not get a correct answer here — it stops being asked. There is one - entry per id because the cache guarantees one instance per id, and re-tracking is an overwrite of a recipe that was - equal anyway.

- -

ReloadOne — 54 lines to 24, and the TODO goes

-
-
-

HotReloadManager.cs:144-197

- 
lock (TrackingLock)
-{
-    if (!Tracked.TryGetValue(id, out candidates)) { return; }
-
-    // TODO: we allow users to add the same asset-id multiple times but we expect
-    // that (after a while) only one of the asset instances is still used by the engine. The
-    // others will be garbage collected eventually. Still, at this point in time we cannot be sure
-    // that the first (or last, or..) alive candidate is the one that will survive.
-    // How should we deal with that?
-    foreach (var candidate in candidates)
-    {
-        if (candidate.IsAlive) { target = candidate; break; }
-    }
-}
-
-// Not finding a target is normal, it means a file
-// changed but the asset depending on it is no longer in use.
-if (target != null) { /* snip: dispatch */ }
-
-
-

proposed

- 
if (!Tracked.TryGetValue(id, out var recipe)) { return; }
-
-// The cache is the authority. A successful lease answers
-// "is anyone still using this?" and guarantees the instance
-// survives until the swap lands.
-if (!recipe.TryStartReload(Cache, FileSystem, PendingReloads, out var reload))
-{
-    Forget(id);   // nobody holds it: drop the recipe and its file edges
-    return;
-}
-
-LogReloadStarted(Logger, id);
-isReloading = true;
-reload.FireAndForget(
-    ex => { LogReloadFailed(Logger, id, ex.SourceException); isReloading = false; },
-    () => { isReloading = false; });
-
-
- -

The recipe — where the lease is taken

- 
public override bool TryStartReload(AssetCache cache, IVirtualFileSystem fileSystem,
-    ConcurrentQueue<ReloadResult> results, out Task reload)
-{
-    // Main thread. Fails when the last bundle already returned the asset,
-    // which is exactly the case where reloading would be pointless or unsafe.
-    if (!cache.TryLease<TAsset>(Id, out var cold))
-    {
-        reload = Task.CompletedTask;
-        return false;
-    }
-
-    reload = Task.Run(async () =>
-    {
-        try
-        {
-            using var stream = new MemoryStream();
-            await AssetEncoder.Encode(Id, Transcoder, Settings, fileSystem, stream).ConfigureAwait(false);
-
-            stream.Seek(0, SeekOrigin.Begin);
-            var hot = await AssetDecoder.Decode(Id, Transcoder, fileSystem, stream).ConfigureAwait(false);
-
-            results.Enqueue(new ReloadResult(Id, hot.BuildMetaData.Dependencies,
-                () => Transcoder.HotSwap(cold, hot.Value)));
-        }
-        catch
-        {
-            // Always enqueue: HotSwapPending is the single place the lease is released.
-            results.Enqueue(ReloadResult.Failed(Id));
-            throw;
-        }
-    });
-
-    return true;
-}
- -

HotSwapPending — the single release point

- 
private void HotSwapPending()
-{
-    while (PendingReloads.TryDequeue(out var result))
-    {
-        try
-        {
-            if (result.PerformHotSwap is not null)
-            {
-                LogHotSwapStarted(Logger, result.Id);
-                result.PerformHotSwap();
-                ReplaceFileDependencies(result.Id, result.Dependencies!);
-                LogHotSwapCompleted(Logger, result.Id);
-            }
-        }
-        catch (Exception ex)
-        {
-            LogHotSwapFailed(Logger, result.Id, ex);
-        }
-        finally
-        {
-            Cache.Return(result.Id);   // the lease from TryStartReload, released exactly once
-        }
-    }
-}
-

One finally, one release point, on the same thread that took the lease. That is the whole lifetime story - and it fits on a screen — which is the property to protect if you build this.

-
-
- -
-
-

The clarity ledger

-

The honest measure of this change is not lines. It is how many uncertain questions the code stops - containing.

- -
- - - - - - - - - - - - - -
Question the code has to answerTodayAfter
Is this asset still in use?WeakReference.IsAlive — GC timing, wrong answer possible (H3)TryLease — deterministic, by definition
Which of N tracked instances is the live one?unanswerable; a 5-line TODO picks the firstquestion deleted — there is one
Is this asset already tracked?broken ReferenceEquals across two types (H5)question deleted — one entry per id
Will the instance survive until the swap?hopethe lease guarantees it
Who tells hot reload that an asset died?nobody — UnTrack has no callernobody needs to; the next lease just fails
Is the tracking table still accurate?grows forever; pruned is computed and discardedpruned on the failed lease
Could the swap target be stale?yes — needs IsCurrent + Target + DiscardNewPartsno — the lease makes it impossible
Who owns the lease, and when is it released?— no lease existsnew: the recipe takes it, HotSwapPending releases it
Does hot reload know about the cache?no — independent objectsnew: yes, a constructor dependency
-
- -

Seven uncertain things out, two certain things in. The two that come in are both of the kind you can - point at in a code review: an ownership rule with one release point, and an edge in the object graph. The seven that - go out are the kind you can only find by reasoning about GC timing, and one of them (H3) is a confirmed defect that - was reproduced last week.

- -
- A structural note about the file-change map -

Dependents stays — a file→ids map is build-graph data, not an asset reference, and this - design does not touch it. But it gains a property it does not have today: it becomes self-pruning. - When TryLease fails, the manager has learned something definite ("this asset is gone") and can drop the - recipe and its file edges on the spot. That is what Forget(id) does above, and it is why the - pruned list can finally be deleted instead of fixed — the code never had a trustworthy signal to - prune on before.

-
-
-
- -
-
-

What this costs

- -
- The real trade -

You are exchanging GC ambiguity for an explicit lifetime obligation that crosses a thread boundary and - a queue. A WeakReference has no obligations: forget about it and nothing bad happens. A lease has - exactly one, and forgetting it is silent until shutdown. That is a good trade — the obligation is local, - checkable, and lives in one finally — but it is a real one and it is worth naming before you start.

-
- -
- - - - - - - -
New obligationWhere it livesIf you get it wrong
Release the lease exactly onceHotSwapPending's finallyRefcount never reaches 0; the asset is never disposed; Cache.Dispose throws “will leak N entries”
Enqueue a result even when the rebuild throwsthe recipe's catch { enqueue; throw; }Same leak, and only on the error path — the one you will not exercise by hand
Drain PendingReloads at shutdownHotReloadManager.DisposeM5 gets worse: a reload in flight at shutdown holds a lease no one will release
-
- -

Two other things get slightly harder

-
    -
  • M5 moves from "undefined" to "actively broken". Today, disposing with a reload in flight is merely - unspecified. With leases, an in-flight reload holds a refcount, so AssetCache.Dispose will report a leak - that is not a real leak. If you build this, M5 stops being a "later" item — at minimum - HotReloadManager.Dispose must drain PendingReloads and return those leases before - Cache.Dispose runs.
    • -
  • The refcount invariant grows a second kind of holder. The invariant proposed for B3/M1 was - “RefCount(id) equals the number of active bundles holding id”. It - becomes “… plus at most one in-flight reload”. Arguably that is cleaner phrased the other - way round — a refcount counts claims, and a reload is a claim — but you have to restate it, and - the XML docs on Return and PutOrLease should say so.
    • -
- -

And one thing that looks like a cost but is not

-

A reload can now finish for an asset the game unloaded halfway through: the lease keeps the entry alive, the swap - runs, then the lease is returned and the asset is evicted and disposed. That is wasted work — one rebuild of - something nobody wants — but it is correct, and it replaces today's behaviour, which is a hot swap onto - an object Collect has already disposed. Wasted work in a dev-only code path on a debounced file change is - not worth engineering away.

-
-
- -
-
-

Size, honestly

-

This is not a size win. Report it to yourself as a certainty win and you will not be disappointed.

- -
- - - - - - - - - - - - -
PieceNowAfterWhy
Track228no list, no dedupe, no TODO
ReloadOne5424candidate loop and its TODO deleted
UnTrack → Forget98same size, finally has a caller
HotSwapPending1921+ the release finally
dependency map1928only if you take the exact-replacement upgrade
HotReloadable → ReloadRecipe4751−weak ref, −IsAlive, +lease and failure enqueue
cache field + ctor03the new edge
Total321~294−27 lines, −8%
-
- -

For comparison, S6 from the first review — hot reload via closures, opt-in behind a flag so shipping builds - never construct it — was −155 lines. These are orthogonal, and this one makes S6 easier: once the - recipe has no asset field it is pure behaviour, which is exactly what a closure is good at. A closure that captures a - transcoder and settings is fine; a closure that captures an asset is the bug you are removing. Do this first, and S6 - becomes a mechanical rewrite instead of a redesign.

- -
- Zero new locking -

Both cache calls happen on the main thread — TryLease in ReloadOne, - Return in HotSwapPending, both reached from AssetManager.Update() and, notably, - outside RequestLock (AssetManager.cs:184-185). So hot reload takes - Cache.Lock and nothing else, no nesting, and the existing - RequestLock → Cache.Lock ordering is untouched. The recommendation from the - previous report to delete TrackingLock and replace it with a debug AssertMainThread is - independent of this change and composes with it — after both, there is very little left for a lock to protect.

-
- -

One invariant also relaxes: the previous report asked you to comment that Cache.Collect() must - run before HotReloadManager.Update(), because evict-then-swap was what made a stale swap detectable. With - leases a stale swap cannot happen at all, so that ordering stops being load-bearing. One fewer thing that breaks - silently if someone reorders two lines.

-
-
- -
-
-

A bonus that falls out

-

Your phrasing — a file that an asset might depend on — is precise, and this design - is what makes the “might” cheap to remove.

- -

Dependents is an over-approximation today because RegisterFileDependencies only ever - adds. An .hlsl that drops an #include keeps that stale edge until process restart, - so touching the removed file triggers a full rebuild forever. With consultation the false positive is cheap when the - asset is dead — one failed TryLease — but it still costs a real rebuild when the asset is - live.

- -

The fix is available now and was not before: a rebuild produces the complete fresh dependency set, and the - recipe can hold the previous one (it is a list of paths, not an asset). So the add becomes a replace:

- - 
private void ReplaceFileDependencies(AssetId id, IReadOnlyList<Dependency> fresh)
-{
-    if (Tracked.TryGetValue(id, out var recipe))
-    {
-        foreach (var old in recipe.Dependencies)
-        {
-            if (Dependents.TryGetValue(old.File, out var ids) && ids.Remove(id) && ids.Count == 0)
-            {
-                Dependents.Remove(old.File);
-            }
-        }
-
-        recipe.Dependencies = fresh;
-    }
-
-    foreach (var dependency in fresh)
-    {
-        // snip: existing add
-    }
-}
- -

About nine extra lines, and Dependents becomes exact instead of monotone. That also closes the residual - noted against M8 in the follow-up report. It is optional — the design works without it — but it is the - cheapest it will ever be, because the recipe already exists and already has nowhere dangerous to point.

-
-
- -
-
-

Roads not taken

- -
- Put the recipe inside AssetCache.Entry — rejected -

The maximal version of the idea: if the cache holds the recipe too, HotReloadManager keeps only - Dependents and its two queues. Tempting, and it would delete Tracked entirely.

-

Rejected because it destroys the thing that makes the cache useful as an authority in the first - place: it is dumb, and it is about lifetime only. Give it transcoder references and it becomes a registry of - everything known about an asset, which means every future question ("what settings? what dependencies? what - version?") has an obvious wrong home. The object? Tag variant is worse still — untyped and - spooky-at-a-distance. Keep the cache answering exactly one question: does anyone still hold this?

-
- -
- Derive the transcoder from the cached instance's runtime type — does not work -

It would be neat if the manager could skip recipes entirely: take entry.Asset.GetType() and look up the - transcoder in AssetManager.Transcoders. Two reasons it fails, both worth knowing so it does not get - re-proposed. Transcoders is keyed on the declared type parameter (IVertexShader), - not the runtime type (VertexShader), so the lookup misses. And settings are per-request and typed, so - even a successful lookup would not tell you what to rebuild with. The recipe object is doing real work; it - is the existential that carries <TAsset, TSettings> across an untyped event.

-
- -
- Keep weak references, add an eviction callback — strictly worse -

This was the previous report's H3 recommendation: have the cache call UnTrack(id) on eviction, keep the - weak reference, and guard the swap with IsCurrent. It works, and it is roughly the same number of lines - (−35 versus −27). But it keeps all three uncertain concepts alive — GC timing, candidate selection, - stale-swap detection — and adds callback plumbing on top. The lease approach removes the questions instead of - answering them. Prefer this design; treat the earlier H3 advice as superseded.

-
-
-
- -
-
-

Verdict

- -
-

Build it — but for the right reason

-

At −27 lines this will not feel like a simplification while you are typing it. It is worth doing because of - what it deletes from the problem rather than from the file: seven questions the current code either answers - wrongly or cannot answer at all, replaced by two it answers by construction. H3 and H5 stop being findings to fix and - become states that cannot be reached, and the two TODOs in HotReloadManager — including the one - that just says this is broken!!!! — are deleted rather than resolved.

-

It also lands at the right time. Hot reload works end to end as of this week, so for the first time there is a - working baseline to compare against: build it, then re-run the two-consecutive-reloads check and the - swap-after-unload check. The second one should flip from swapped=True (today's bug) to - swapped=False, and that single assertion is the whole design in one line.

-

Order: finish B3 + M1 first — the lease arithmetic has to be right before you add a second - kind of lease holder. Then this. Then M5, which this change promotes from "later" to "required". Then M8's - TrackingLock deletion, which by that point is nearly free.

-
-
-
- -
-
-

Scope. A design exploration, not a review — nothing here is implemented and no code was - changed. Line counts are estimates against the current HotReloadManager.cs (274 lines) and - HotReloadable.cs (47 lines) at c7517e7. The one load-bearing claim — that - TryLease fails deterministically once the last user returns an asset — is read directly from - AssetCache.Return at lines 93–99, where Entries.Remove happens immediately and only the - Dispose is deferred to Collect.

-

Relationship to the other reports. Supersedes the H3 recommendation in - AssetPipelineVNextReviewContinued.html (cache-as-authority via eviction callback), which reached a similar - place by a worse route. Independent of, and compatible with, the B3/M1 lease-accounting fix and the M8 - TrackingLock deletion in that same report, and it makes S6 from - AssetPipelineVNextReview.html easier rather than harder.

-
-
- - - diff --git a/Research/AssetPipelineLoadingGroups.md b/Research/AssetPipelineLoadingGroups.md deleted file mode 100644 index d5413b2..0000000 --- a/Research/AssetPipelineLoadingGroups.md +++ /dev/null @@ -1,298 +0,0 @@ -# Asset Pipeline — Loading Groups - -Status: sketch for discussion, 2026-08-10. Companion to `AssetPipelineArchitecture.md`. - -## Context - -While working on `source/CapriKit.AssetPipeline/vNext/` — the threading-aware rewrite. `AssetCache` -was already done (thread-safe lease/return with deferred disposal); the open question was how -assets get *delivered* now that `Load` no longer returns a `Task` to gameplay code. - -The concrete worry: in the old engine a `Car` took its texture, model and shader through -constructor injection. With an optimistic (`TryLoad`) or channel-based model, every system would -seem to need a "do I have everything yet?" check in `Update`, every frame. That is the thing this -document is trying to avoid. - -## The framing - -The question isn't *task vs channel vs optimistic*. It's **who is allowed to exist before their -assets exist**. Engines don't answer that uniformly — they split the world: - -- **Engine assets** — lighting/shadow shaders, BRDF LUT, blue noise, error textures. Part of the - engine build, not content. Must exist or the engine is broken. Loaded in a boot phase *before* - any system is constructed. Constructor injection, no nulls, no polling. -- **Content assets** — levels, entities, characters. Their consumers have to tolerate absence - anyway (streaming, LOD, missing file, hot reload), so absence is designed in rather than bolted on. - -Source calls this `Precache*` at level load — requesting a non-precached asset at runtime is a hard -error in dev builds. Unreal splits `FStreamableManager` requests from always-loaded cooked packages. - -The rule that falls out: **never expose "the asset might not be here" to gameplay code.** That is -precisely what forces a per-frame check into every system. `TryLoad` is fine as an internal -cache-hit fast path; it is a trap as the public API. - -## Options considered - -| Option | Shape | Good for | Bad for | -|---|---|---|---| -| **A** Resolve-then-construct | Boot phase batch-loads declared requirements in parallel, systems are constructed from a resolved lookup that cannot fail | Lighting, Shadowing, engine systems | Anything streamed — loading the whole game up front | -| **B** Placeholder + `HotSwap` | `Load` returns a usable instance immediately whose contents are a placeholder; swapped in place when the payload lands | Assets whose absence only affects pixels | Simulation data (placeholder collision mesh = fall through floor), and **shaders** — see below | -| **C** Group barrier + completion channel | Interlocked counter per group; the worker that drives it to zero pushes the group onto a channel the main thread drains | Levels, entities, anything with N dependencies | Nothing much — it is the general mechanism | -| **D** Async confined to load scopes | Workers `await` N assets, construct the finished object, hand it to the sim through one channel | Level loading, streaming coordinators | — | - -Tasks were never the mistake; handing a `Task` to *gameplay* was, because async then infects -everything upward. In D, `Car`'s constructor still takes its assets directly — it just isn't the -main thread calling it. - -**Recommended split for CapriKit:** A for engine systems, B for renderer content, C+D for levels -and entities. - -## Why shaders can't use placeholders (Option B) - -A placeholder is a valid substitute only when **every instance of the asset type shares one -interface**. A texture qualifies: an SRV is an SRV, the consumer doesn't care what pixels are -behind it. Content varies, interface doesn't. - -For a shader the interface *is* the identity: - -- `CreateInputLayout` validates the `InputElementDescription[]` against the VS input signature in the blob -- cbuffer register assignments (`b0`, `b1`, …) and their field layouts -- SRV/sampler slots (`t0`, `s0`) - -A placeholder VS substitutes for a real VS only if it has the same input signature *and* binding -layout — at which point it is a hand-written stub per shader, not a placeholder. - -### The same gap is a latent hot-reload bug - -`IVertexShader.HotSwap` (`source/CapriKit.DirectX11/Resources/Shaders/VertexShader.cs:12`) swaps -`Blob` and `ID3D11VertexShader`. But the `IInputLayout` was created from the *old* blob and is owned -by the consumer — e.g. `ImGuiEffect.cs:24` holds it in a separate field. Nothing tells that layout -its shader changed. - -D3D11 won't crash: `CreateInputLayout` validates at creation and the layout is independent -afterwards. But reload a shader whose signature grew a `TEXCOORD1` and the old layout no longer -feeds it — debug-layer complaint, zero/garbage in that register. Subtly wrong, which is worse. - -`HotSwap` on a bare vertex shader is incoherent *because* a bare vertex shader isn't independently -usable — the same premise as the placeholder problem. - -### Fix: the effect is the atomic asset - -The transcoder's `TAsset` should be the smallest **self-consistent** unit — the thing owning a -complete, valid interface: - -```csharp -internal sealed class Effect // IAssetTranscoder produces this -{ - public ID3D11VertexShader Vertex { get; private set; } - public ID3D11PixelShader Pixel { get; private set; } - public IInputLayout InputLayout { get; private set; } // built from *this* blob, at decode - - // snip -} - -public override void HotSwap(Effect instance, Effect newParts) -{ - var old = instance.Exchange(newParts); // shader + layout swap together, main thread - // snip: dispose old -} -``` - -A mismatched (blob, layout) pair becomes unrepresentable — the layout is created inside `Decode` -from the blob being decoded. `CapriKit.Generators.HLSL` already emits the input element -descriptions as compile-time constants, so this construction is deterministic. - -Shaders then belong in **category A**: they're a few KB, the set is closed at compile time (the -generator enumerates every `#pragma VertexShader` / `#pragma PixelShader` entry point, so the boot -manifest can be generated), and a renderer missing its lighting shader isn't degraded, it's -non-functional. - -For content-driven materials the answer is **skip the draw**, not substitute: the renderable isn't -registered with the render system until its effect exists. Nothing to branch on in the hot loop — -the object simply isn't in the visible set. That's UE5's PSO-precache behaviour, and it composes -with the group barrier below. - -A real shader fallback is achievable but only under a convention: fix the binding layout -engine-wide (per-frame `b0`, per-view `b1`, per-object `b2`) and standardize vertex formats. Then a -magenta error shader with a matching input signature is a genuine drop-in — that is exactly why -Unity's error shader works, it demands only the minimal interface. Worth building for the -**broken/corrupt** case so a bad edit doesn't kill the frame; not worth it for the not-yet-loaded -case, which boot-loading eliminates. - -## Option C sketch — typed loading groups - -Assumption: requirements are hardcoded in C# for the foreseeable future. - -### The type-safety trick - -Make "not loaded yet" **unrepresentable** in the consumer's types. Three types do it: - -- `Ticket` — a claim check with *no value accessor at all* -- `ResolvedAssets` — the only way to redeem a ticket, obtainable only after the group completes -- `AssetGroup` — produces one strongly-typed result whose fields are plain non-nullable assets - -No `IsLoaded`, no `.Value` that can be read early, and no arity explosion -(`AssetGroup`) because tickets carry their type individually and a closure -reassembles them. - -### Declaration side - -```csharp -public abstract class Ticket -{ - internal AssetId Id { get; init; } - internal object? Asset; // written by loader, read on main thread — the one erasure point -} - -/// A claim check for one asset. Redeem with . -public sealed class Ticket : Ticket where TAsset : class; - -public readonly struct ResolvedAssets -{ - // Safe by construction: the ticket's loader produced exactly TAsset - public TAsset Get(Ticket ticket) where TAsset : class - => (TAsset)ticket.Asset!; -} - -public sealed class AssetGroupBuilder -{ - public Ticket Add( - AssetId id, IAssetTranscoder transcoder, TSettings settings) - where TAsset : class - { - var ticket = new Ticket { Id = id }; - // Both type params erased into the closure; TAsset survives on the ticket - Requests.Add(new Request(ticket, (m, ct) => m.LoadAsync(id, transcoder, settings, ct))); - return ticket; - } - - public AssetGroup Build(Func factory) { /* snip */ } -} -``` - -### A hardcoded requirement list - -```csharp -internal sealed class ShadowAssets -{ - public static AssetGroup Define( - AssetGroupBuilder b, EffectTranscoder effects, TextureTranscoder textures) - { - var shadow = b.Add("shaders/shadow.hlsl", effects, EffectSettings.Default); - var noise = b.Add("textures/blue-noise.png", textures, TextureSettings.Default); - - return b.Build(r => new ShadowAssets(r.Get(shadow), r.Get(noise))); - } - - private ShadowAssets(Effect shadow, ITexture2D noise) - { - Shadow = shadow; - Noise = noise; - } - - public Effect Shadow { get; } // non-nullable, always valid - public ITexture2D Noise { get; } -} - -// The system never sees the pipeline at all -internal sealed class ShadowSystem(ShadowAssets assets, Device device) { /* snip */ } -``` - -Adding a requirement is two lines and the compiler forces the constructor to match. Deleting one -breaks the build. That is the payoff for hardcoding requirements. - -### The barrier and the drain - -```csharp -public abstract class AssetGroup : IDisposable -{ - private int outstanding; - - // Called on whichever worker finished the request - internal void OnRequestCompleted(ChannelWriter ready) - { - if (Interlocked.Decrement(ref outstanding) == 0) { ready.TryWrite(this); } - } - - internal abstract void Materialize(); // main thread only -} - -public sealed class AssetGroup : AssetGroup -{ - private readonly Func Factory; - private readonly TaskCompletionSource Source = - new(TaskCreationOptions.RunContinuationsAsynchronously); // don't hijack the main thread - - internal override void Materialize() => Source.SetResult(Factory(new ResolvedAssets())); - - public Task Completion => Source.Task; - public bool TryTake([NotNullWhen(true)] out TResult? value) { /* snip */ } -} -``` - -```csharp -// AssetManager.Update() — main thread, once per frame -while (ReadyGroups.Reader.TryRead(out var group)) -{ - group.Materialize(); // usually zero iterations -} -Cache.Collect(); -``` - -This answers the original worry: the main thread drains **one** channel of finished groups. No -system polls its own assets, and the per-frame cost is O(groups that finished this frame), normally -zero. - -One mechanism serves both consumption styles: - -```csharp -// Boot / loading screen (Option A) -var group = ShadowAssets.Define(builder, effects, textures); -services.AddSingleton(new ShadowSystem(await group.Completion, device)); - -// Streaming (Option C) — entity doesn't exist until the group lands -if (pendingLevel.TryTake(out var level)) { World.Install(level); } -``` - -### Lifetime and failure - -Make the **group** the refcount unit, not the individual asset — it lines up with -`AssetCache.Return` and makes level unload a single `Dispose`: - -```csharp -public void Dispose() -{ - foreach (var ticket in Tickets) - { - if (ticket.Asset is not null) { Cache.Return(ticket.Id); } - } -} -``` - -The failure path matters because `AssetCache.Dispose` throws on leaks. If request 4 of 5 throws, -the three that already landed hold leases. Catch per-request, store the exception, let the counter -reach zero anyway, and have `Materialize` fault the group *and* return the partial leases. -Otherwise one bad asset file becomes a leak exception at shutdown that says nothing about the cause. - -## Constraints to document for transcoder authors - -**Where `Decode` may run.** `Device.cs:21-31` creates the device without -`DeviceCreationFlags.Singlethreaded`, so `ID3D11Device` resource creation is free-threaded — -`CreateVertexShader` / `CreateTexture2D` on a worker is fine. `ID3D11DeviceContext` is **not**. A -transcoder needing `Map` or `UpdateSubresource` must either create immutable resources with initial -data (no context involved) or defer that step to `Materialize` on the main thread. Violating this -produces corruption rather than an exception, so it needs to be an explicit written rule. - -**Ticket provenance.** Nothing at compile time stops redeeming a ticket from group A inside group -B's factory. The closure capture makes it awkward in practice; a `Debug.Assert(ticket.Owner == this)` -in `Get` covers the rest cheaply. - -## Open questions - -- Does `AssetGroup` need `Task` at all, or is `TryTake` + a main-thread callback enough? - `Completion` is convenient for boot code that legitimately awaits on a loading screen. -- Where does hot reload re-enter? A reloaded asset inside a live group needs `HotSwap`, not - re-materialization — the `TResult` already handed out must keep its identity. -- Should `AssetManager.TryLoad` stay public? Current stub returns `bool` with nowhere to put the - asset; it wants `out TAsset` with `[NotNullWhen(true)]`, mirroring `AssetCache.TryLease` — and - arguably it should be internal, as the fast path the three policies sit on. diff --git a/Research/AssetPipelineLoadingGroupsV2.md b/Research/AssetPipelineLoadingGroupsV2.md deleted file mode 100644 index 7aeaf2b..0000000 --- a/Research/AssetPipelineLoadingGroupsV2.md +++ /dev/null @@ -1,273 +0,0 @@ -# Asset Pipeline — Loading Groups V2 (bundles and promises) - -Status: design correction, 2026-08-11. Supersedes the Option C sketch in -`AssetPipelineLoadingGroups.md`; that document's framing (engine vs content assets, why shaders -can't use placeholders, the transcoder threading rules) still stands. - -## Context - -Continuing `source/CapriKit.AssetPipeline/vNext/`. The V1 sketch used `Task`/`await` for group -completion. That was dropped in favour of two explicit consumption styles: - -- **block** — assets the caller cannot function without (bootstrap, engine systems) -- **poll** — assets the caller can proceed without, so the main thread keeps running a loading screen - -`AssetGroup` became `AssetBundle`, `Ticket` became `Promise`, and `AssetGroupBuilder` was -dropped in favour of `AssetManager.Load` returning promises directly. - -That last change is what caused the deadlock in the design: - -```csharp -// AssetBundle.cs — the shape that doesn't work -var a = assetManager.Load(id, default); -var b = assetManager.Load(id, default); -return assetManager.Bundle(r => new ExampleAssets(r.Get(a), r.Get(b))); -``` - -The bundle needs the promises to exist first (they're captured in the resolver closure), but -`AssetBundle.OnRequestCompleted` means each promise needs a back-pointer to the bundle. And if a -load finishes before `Bundle(...)` is called, `Update()` has nothing to signal — the countdown -never reaches zero. - -## Diagnosis - -Three problems were tangled into one. Only the first is about ordering, and it is self-inflicted. - -### 1. The cycle exists only because the bundle wants to be *told* - -`OnRequestCompleted` is a push notification. Push requires the observer to exist before the event. -Delete the notification and the ordering constraint goes with it — the bundle reads the state of its -promises instead of counting events: - -```csharp -public bool Check([MaybeNullWhen(false)] out T value) -{ - if (this.result is null) - { - foreach (var promise in this.Promises) - { - if (!promise.IsResolved) { value = default; return false; } - } - this.result = Factory(new PromiseResolver(this)); // exactly once - } - value = this.result; - return true; -} -``` - -What this removes: - -- **The chicken-and-egg.** Promises never reference the bundle while loading. Build order is free. -- **"Already loaded before the bundle existed."** A promise resolved at birth from the cache is - simply resolved; the scan sees `true`. There is no missed edge to compensate for, because the - bundle reads a *state*, not a stream of events. -- **`CountdownEvent`, `OnRequestCompleted`, and the `Bundle` half of the `OutstandingPromises` - tuple** (`AssetManager.cs:8`). - -Cost is O(n) per `Check` with n = 2–20 promises per bundle, against O(1) for the counter. Worth it. -If bundle counts ever grow into the thousands, reintroduce the counter — but it must then be -initialised at *build* time from the promises that are not yet resolved, which is the same -reconciliation done eagerly. - -### 2. The in-flight table is 1:1, but the relationship is 1:N - -`Dictionary` holds one promise per id. Two bundles can want the -same asset, and one bundle can want it twice — `ExampleAssets.Define` requests -`new AssetId("key", "path")` twice today, so the second `Load` would overwrite the first entry and -promise `a` would hang forever. This was the `// TODO: what if 2 promises wait for the same thing`. - -Make the in-flight table the dedupe point: - -```csharp -private readonly Dictionary> InFlight = []; - -internal Promise Request(AssetId id, TSettings settings) - where TAsset : class -{ - var promise = new Promise { Id = id }; - - if (Cache.TryLease(id, out var cached)) - { - promise.Value = cached; // resolved before it was ever outstanding - return promise; - } - - if (this.InFlight.TryGetValue(id, out var waiting)) { waiting.Add(promise); } - else { this.InFlight[id] = [promise]; Dispatch(id, settings); } - - return promise; -} - -public void Update() -{ - while (Ready.TryRead(out var kv)) - { - if (!this.InFlight.Remove(kv.Id, out var waiting)) { continue; } - - var asset = Cache.PutOrLease(kv.Id, kv.Asset); - foreach (var promise in waiting) { promise.Value = asset; } - // snip: take waiting.Count - 1 extra leases - } - Cache.Collect(); -} -``` - -Two requests for one id now cost one disk read and fill both promises. - -**Refcount detail:** `AssetCache.PutOrLease` takes exactly one lease. N promises that will each -`Return` once on bundle disposal need N−1 additional `TryLease` calls here, or `AssetCache.Dispose` -throws on the asymmetry. - -### 3. The builder from V1 was dropped and is still needed - -`AssetGroupBuilder` in V1 gave the group an identity before its tickets. Moving `Load` onto the -manager is what made the ordering feel impossible. Pull (fix 1) already resolves the ordering, but -the builder is still wanted for three other reasons: it defines which promises a bundle **owns** -(for `Dispose` → `Cache.Return`), it supplies the resolver-ownership check, and it makes "when does -loading start" explicit. - -```csharp -public sealed class AssetBundleBuilder(AssetManager manager) -{ - private readonly List Promises = []; - - public Promise Load(AssetId id, TSettings settings) - where TAsset : class - { - var promise = manager.Request(id, settings); - this.Promises.Add(promise); - return promise; - } - - public AssetBundle Build(Func factory) where T : notnull - { - var bundle = new AssetBundle(manager, [.. this.Promises], factory); - foreach (var promise in this.Promises) { promise.Owner = bundle; } - return bundle; - } -} -``` - -```csharp -public static AssetBundle Define(AssetManager assetManager) -{ - var builder = assetManager.CreateBundle(); - var a = builder.Load(new AssetId("a", "example.txt"), default); - var b = builder.Load(new AssetId("b", "example.txt"), default); - - return builder.Build(r => new ExampleAssets(r.Get(a), r.Get(b))); -} -``` - -Loading starts at `Load`, not at `Build` — there is no reason to defer, because an early completion -is now harmless. `Owner` is assigned in `Build`, which closes V1's open question on promise -provenance: `PromiseResolver` takes the bundle (`new PromiseResolver(this)`) and `Get` asserts -`promise.Owner == owner` rather than the commented-out check in `Promise.cs:15`. - -## The deadlock that hasn't been hit yet - -Independent of the above, and the most dangerous item here because it only appears once `Update` is -wired into the game loop. - -`AssetBundle.Wait` blocks on `Outstanding.Wait()`. `OnRequestCompleted` fires from -`AssetManager.Update()`. In the bootstrap case both are the main thread: it blocks waiting for a -signal only it could deliver. - -This forces a decision about **who applies completions**. Keep it on the main thread — that gives -lock-free `Promise.Value` and `InFlight`, and it is where a transcoder's `ID3D11DeviceContext` -finalisation step has to run anyway (see the transcoder constraints in V1). Blocking then means -*pumping*, not sleeping: - -```csharp -public T Wait(CancellationToken cancellationToken = default) -{ - var spin = new SpinWait(); - while (!Check(out var value)) - { - cancellationToken.ThrowIfCancellationRequested(); - Manager.Update(); // the waiting thread does the work instead of sleeping on it - spin.SpinOnce(); - } - return value; -} -``` - -This is the helper pattern from job systems (Unity's `JobHandle.Complete`), and it collapses both -consumption styles onto one mechanism: `Wait` is `Check` in a pumping loop, `Check` is one test -against a drain the game loop already performed. No `await`, no `CountdownEvent`, no kernel wait. - -**Constraint to document:** `Wait` and `Check` are main-thread only. If a worker should later build -a level off-thread (V1 option D), give that path a real `ManualResetEventSlim` signalled from -`Update`. - -## Two bugs in the current code, independent of the redesign - -**`Check` re-materialises the bundle.** `AssetBundle.cs:47` calls `Wait()`, which calls -`Resolver(...)`, so every successful `Check` constructs a *new* `ExampleAssets`. Polling it for 60 -frames on a loading screen yields 60 distinct instances, and hot reload has no stable object to swap -into. Materialise once and cache (`this.result` above) — this is what V1's `Materialize()` was for. - -**No faulted state.** `Promise` has only `Value`. Give it an `ExceptionDispatchInfo?` alongside, and -treat faulted as resolved. Otherwise one missing file makes `Wait` spin forever instead of throwing, -and the promises that did land leak their leases into `AssetCache.Dispose`. V1 already called this -out ("let the counter reach zero anyway"); with pull it becomes "let the promise resolve as -faulted". - -## Nullability footnote on `Check` - -The original signature `bool Check([NotNullWhen(true)] out T? value)` fails to compile under this -repo's `nullable` (`Directory.Build.props:12`) with **CS8762**, -*"Parameter 'value' must have a non-null value when exiting with 'true'"* — verified against the -SDK. - -For an unconstrained type parameter, `T` and `T?` have the same null-state: `T` could be -instantiated as `string?`, so the compiler cannot prove `Wait()` returns non-null. `[NotNullWhen]` -is a promise with nothing to back it. Either use the canonical `TryGetValue` shape -`[MaybeNullWhen(false)] out T value` (no constraint needed), or keep `where T : notnull` on -`AssetBundle`, which the current code already has and which makes the original signature legal. -`notnull` is worth keeping regardless — a resolved bundle that is null is meaningless. - -## Resulting data model - -```csharp -public abstract class Promise -{ - internal AssetId Id { get; init; } - internal object? Value; // written on the main thread only - internal ExceptionDispatchInfo? Error; // faulted counts as resolved - internal AssetBundle? Owner; // assigned in Build, for the provenance check - internal bool IsResolved => Value is not null || Error is not null; -} - -public abstract class AssetBundle; // no counter, no OnRequestCompleted - -public sealed class AssetBundle : AssetBundle where T : notnull -{ - private readonly AssetManager Manager; - private readonly Promise[] Promises; // also the Dispose → Cache.Return set - private readonly Func Factory; - private T? result; // materialised once - // snip: Check, Wait, Dispose -} - -public sealed partial class AssetManager -{ - private readonly Dictionary> InFlight = []; - private readonly LightweightChannel<(AssetId Id, object Asset)> Ready = new(); - private readonly AssetCache Cache = new(); - // snip: CreateBundle, Request, Update, Dispose -} -``` - -## Open questions - -- Cancellation: dropping a bundle mid-load leaves entries in `InFlight` whose promises nobody reads. - Harmless (the drain skips them) but it means the load still completes and takes a lease. Does - `AssetBundle.Dispose` need to prune `InFlight`, or is letting it land and immediately `Return` it - simpler? -- Hot reload re-entry is unchanged from V1 and still unanswered: a reloaded asset inside a live - bundle needs `HotSwap`, not re-materialisation, because `this.result` has already been handed out. -- Does `TryLoad` (`AssetManager.cs:12`) survive at all? With `Request` checking the cache inline it - looks redundant — V1 already suspected it should be internal. -- `SpinWait` in the `Wait` pump busy-burns a core during bootstrap. Probably fine for a few hundred - ms at startup; revisit if boot loading grows. diff --git a/Research/AssetPipelineReview.md b/Research/AssetPipelineReview.md deleted file mode 100644 index 650120c..0000000 --- a/Research/AssetPipelineReview.md +++ /dev/null @@ -1,292 +0,0 @@ -# Asset Pipeline Review - -> Multi-agent review of `source/CapriKit.AssetPipeline` (branch `feature/asset_pipeline`), -> triggered by a worry that the caching, hot-reloading and `AssetJob` handling were -> too complex and mixed too many paradigms. Three review agents (correctness, -> clarity/paradigm-consistency, simplification) assessed the source without running any -> tools; every finding below was cross-checked against a manual read of the code. - -**Headline:** the worry was right for a concrete reason. The non-exhaustive `AssetJob` -consumption API (`OnSuccess`/`OnFailure`/`OnMissing`) is not just noisy — it *directly -caused* three correctness bugs by letting success paths fall through to `throw`. The -paradigm problem and the correctness problem are the same problem. - -## Overview - -| # | Cluster | Finding | Severity | Location | -|---|---------|---------|----------|----------| -| F1 | A · Result API | Successful full build never returns — falls through to `throw` | Critical | `AssetManager.cs:94-105` | -| F2 | A · Result API | Rebuild failure check inspects the **wrong** job variable | High | `AssetManager.cs:100` | -| F3 | A · Result API | Successful hot-swap falls through to `throw` (logged as failure) | High | `HotSwappable.cs:18-32` | -| F4 | C · Lifetime | `PopScope` mutates the dictionary while enumerating it | High | `AssetMemoryCache.cs:67-74` | -| F5 | B · Errors | `OpenRead` + `checked` cast throw *outside* the catch, escaping the `Failure` contract | Low | `AssetDecoder.cs:24-25` | -| F6 | A · Result API | `AssetJob` offers 4 consumption modes; the `On*` trio is non-exhaustive | High | `Asset.cs:48-94` | -| F7 | B · Errors | Five error paradigms; an exception is captured -> ferried -> re-thrown | High | decoder -> manager | -| F8 | B · Errors | The `Failure`/EDI state is probably unnecessary | Medium | `Asset.cs`, `AssetManager.cs` | -| F9 | A · Simplify | `AssetFileCache.Load`'s `Match` is an identity on 2 of 3 arms | Medium | `AssetFileCache.cs:14-25` | -| F10 | A · Simplify | `SettingsEqual` re-serializes both sides on every load | Medium | `AssetFileCache.cs:48-57` | -| F11 | A · Simplify | The "success->register / else rethrow / else throw" tail is duplicated | High | `AssetManager.cs` + `HotSwappable.cs` | -| F12 | C · Lifetime | Two ownership models (scope-stack vs `WeakReference`); pop doesn't untrack | High | `AssetMemoryCache` + `HotSwapManager` | -| F13 | C · Threading | `Track` mutates plain dictionaries off the main thread | Medium | `HotSwapManager.cs:49-53` | -| F14 | D · Naming | "Job", `On*`, and dual "Cache"/"Load" each name two things | Medium | cross-cutting | -| F15 | D · API surface | Public API leaks `Encode`/`Decode`/`AssetJob` internals | Medium | `AssetManager`, `Asset.cs` | -| F16 | D · API surface | `IAssetTranscoder` mixes public/internal members across two arities | Low | `IAssetTranscoder.cs` | - -The four clusters map onto the three original worries: **A + B = the `AssetJob` -handling**, **C = caching + hot reloading**, **D = the cross-cutting naming/surface tax**. -Fix A first — it contains the only shippable blockers. - ---- - -## Cluster A — The result API, and the bugs it caused - -`AssetJob` lets you consume a three-state value with a **non-exhaustive** idiom -(`if (job.OnSuccess(out …)) { … }`), which silently collapses the other two states into a -fall-through. Nothing forces you to handle all three, so a missing `return` compiles -cleanly and ships. - -### F1 — Successful build never returns *(Critical)* -`AssetManager.cs:94` - -```csharp -var getFromFullBuild = await Decode(id); -if (getFromFullBuild.OnSuccess(out var freshAsset)) -{ - Cache.Add(id, freshAsset.Value); - HotSwapManager.Track(freshAsset, settings); - // no return — control falls through … -} -// … -throw new Exception($"Asset {id} could not be found"); // reached on success -``` - -**Failure scenario:** memory miss + disk miss + a *successful* build -> the asset is -built, cached and tracked, then `Load` throws "could not be found". (A second call would -hit the memory cache and succeed, making the bug maddening to diagnose.) Every first-time -load of a not-yet-built asset throws. - -**Fix:** add `return freshAsset.Value;` inside the success block. - -### F2 — Wrong job variable in the failure check *(High)* -`AssetManager.cs:100` - -```csharp -if (getFromFileCache.OnFailure(out var rebuildFailure)) { rebuildFailure.Throw(); } -``` - -After the rebuild, this inspects the **original file-cache** job, not `getFromFullBuild`. -So a genuine *build* error is discarded (you get the generic "could not be found"), while a -stale *cache* error can be re-thrown even though the rebuild is what actually ran. -**Fix:** check `getFromFullBuild.OnFailure(...)`. This also proves F8 — the "remember the -cache error and rethrow it later" behaviour isn't actually relied upon. - -### F3 — Successful hot-swap throws *(High)* -`HotSwappable.cs:18` - -Same shape: the `OnSuccess` block does the swap and re-tracks, but doesn't `return`, so it -falls to `throw new Exception($"Asset {Id} … could no longer be found")` on line 32. That -exception is caught by `HotSwapManager.HotSwapCompleted` and logged via `LogReloadFailed` — -so **every successful hot reload is reported as a failure**, after its side effects already -ran. **Fix:** add `return;` after the success block. - -### F6 — Four ways to consume one value; two are unsafe *(High)* -`Asset.cs` - -`AssetJob` exposes `OnSuccess`/`OnFailure`/`OnMissing` (imperative, **non-exhaustive**) -*and* two `Match` overloads (functional, **exhaustive**). Offering both means every reader -must learn two APIs and every author must pick the safe one unaided — and F1-F3 are what -happens when they don't. Secondary smell: `OnSuccess(out Asset? asset)` returns a -**nullable** even on success, so a caller who trusts the out over the bool invents yet -another path. - -**Recommendation — keep the exhaustive one.** The whole point of the tri-state is to *not* -lose the Failure/Missing distinction, and only `Match` (or a `switch` on an explicit state -enum) enforces that. - -- Delete the `On*` trio. Consume via `Match`, or add - `enum AssetJobState { Success, Failure, Missing }` + a non-nullable payload accessor and - `switch` on it — a `switch` expression still gets exhaustiveness warnings; three - independent `bool`s never can. -- Make the success payload non-nullable. - -### F11 — Unify the duplicated tail *(High)* -The "on success add/track/use; else rethrow EDI; else throw not-found" shape is hand-rolled -in **both** `AssetManager.Load` and `HotSwappable.HotSwap` — which is exactly why the same -fall-through slipped into both. Extract it once so it can't recur: - -```csharp -private TAsset Register(Asset asset, IAssetSettings settings) where T : class -{ - Cache.Add(asset.Id, asset.Value); - HotSwapManager.Track(asset, settings); - return asset.Value; -} - -[DoesNotReturn] -private static TAsset Unavailable(AssetJob job, AssetId id) where T : class -{ - if (job.OnFailure(out var edi)) { edi.Throw(); } - throw new AssetNotFoundException(id); -} -``` - -The `Load` tail then becomes -`return getFromFullBuild.OnSuccess(out var fresh) ? Register(fresh, settings) : Unavailable(getFromFullBuild, id);` -— F1 and F2 both become structurally impossible. - -### F9 — Dead `Match` ceremony *(Medium)* -`AssetFileCache.Load` uses a 3-arm `Match` where the failure and missing arms both just -`return job` — only the success arm has logic. Collapse to early returns: - -```csharp -var job = await AssetDecoder.Decode(id, transcoder, FileSystem); -if (job.OnSuccess(out var asset)) - return IsUpToDate(asset) && SettingsEqual(transcoder, asset.Settings, settings) - ? job : AssetJob.Missing(id); -return job; // failure and missing pass straight through -``` - -### F10 — `SettingsEqual` serializes twice per load *(Medium)* -It writes both the embedded and requested settings into fresh `ArrayBufferWriter`s -just to compare bytes — two allocations + two serializations on every disk hit. If settings -are `record`/`record struct`, compare structurally instead: - -```csharp -// default method on IAssetTranscoder -bool SettingsEqual(TSettings a, TSettings b) => EqualityComparer.Default.Equals(a, b); -``` - -Cheaper fallback if you keep bytes: `AssetDecoder` already read the settings bytes off disk -— stash them on the `Asset` and serialize only the *requested* side once. - ---- - -## Cluster B — Too many error paradigms - -### F7 — Five vocabularies, and a full round-trip *(High)* -The module speaks (1) plain `throw`, (2) `ExceptionDispatchInfo` capture-and-rethrow, -(3) the tri-state `AssetJob`, (4) `bool`-try (`TryGet`, the `On*` trio), and (5) nullable -`out`. They don't layer — they convert into each other in a circle: `AssetDecoder` catches -**every** exception and demotes it to `Failure(EDI)`; that failure rides through -`AssetFileCache`, is held across an `await` while `AssetManager` re-encodes, and is finally -re-thrown at the bottom of `Load`. Tracing "what happens on a corrupt file?" means holding -the demoted error in your head across the whole method. - -**Fix:** make `AssetJob` the *only* currency inside the pipeline and convert to an exception -exactly once, at the single public `Load` seam — by `Match`-ing the **final** job (not -re-reading a stale one). Keep `ExceptionDispatchInfo` **only** where it earns its keep: -preserving a stack trace across the `await`/thread-pool hop in hot reload. Everywhere the -exception never crosses a thread, store a plain `Exception`. - -### F8 — The `Failure` state may be dead weight *(Medium)* -`Failure` exists solely to carry an EDI so `Load` can *defer* a rethrow — but F2 shows that -deferral is buggy and unrelied-upon, and `Missing` vs `Failure` are otherwise treated -identically (both fall through to rebuild). Consider collapsing to **two states + -exceptions**: `Decode` returns `Asset?` (`null` = missing) and simply *throws* on a -corrupt file; `Load` wraps the disk read in `try/catch`, logs, and rebuilds. That deletes -the `Failure` state, the EDI field, `OnFailure`, and all the remember-then-rethrow -bookkeeping. This is the single biggest reduction in paradigm count — decide first whether -you ever genuinely need "surface the *cache* error only when the *rebuild* also fails" (the -current code suggests not). - -### F5 — Decoder error-path gap *(Low)* -`AssetDecoder.cs:24-25`: `fileSystem.OpenRead(...)` and `checked((int)input.Length)` run -*before* the `try` (line 28), so an IO/sharing error or a >2 GB `OverflowException` escapes -as a raw throw instead of the `Failure` job the method otherwise promises. **Fix:** move -both inside the `try` (or accept the inconsistency once F8 makes throwing the norm). - ---- - -## Cluster C — Lifetime & threading - -### F4 — `PopScope` crashes and leaks *(High)* -`AssetMemoryCache.cs:67` - -```csharp -foreach (var (key, value) in Cache) { if (value.Scope >= scope) { value.Disposable?.Dispose(); Cache.Remove(key); } } -``` - -`Dictionary` bumps its version on `Remove`, so the next `MoveNext` throws -`InvalidOperationException: Collection was modified` — on essentially every real scope pop -that held an asset. The loop then aborts, so the remaining assets in that scope are never -disposed (leak). **Fix:** snapshot the keys first: - -```csharp -foreach (var key in Cache.Where(kv => kv.Value.Scope >= scope).Select(kv => kv.Key).ToList()) -{ - Cache[key].Disposable?.Dispose(); - Cache.Remove(key); -} -``` - -### F12 — Two ownership models that can disagree *(High)* -`AssetMemoryCache` says the **cache owns** assets (holds `IDisposable?`, disposes on pop). -`Reloadable` says they're **not owned** (holds only a `WeakReference`, treats -"collected" as normal). Both can't be the authority on "is this alive?" Because the cache -strongly holds the disposable for the whole scope, the weak-ref can't die *until* pop — and -crucially **nothing untracks the `Reloadable` from `HotSwapManager` on `PopScope`**. So -after a pop, `Tracked`/`Dependents` still reference the now-disposed asset; a file change -can hot-swap a **disposed instance** (this is the `// TODO: cold can be alive but disposed`). - -**Fix:** make **scope the single owner**. Have `Reloadable`/hot-reload hold the *AssetId* -(a cache key) instead of a `WeakReference`, and look the live instance up in the -cache at reload time — reloads then naturally stop when the scope is popped. Add an -`Untrack(id)` called from `PopScope`. Reserve `WeakReference` only if you deliberately -support caller-owned assets the cache does *not* dispose (and document that split). - -### F13 — `Track` races the main thread *(Medium)* -`HotSwapManager.Track` writes plain `Dictionary`s (`Tracked`, `Dependents`). It's documented -main-thread, but `AssetManager.Load` calls it *after* `await`s (lines 78, 94); with no -game-loop `SynchronizationContext` those continuations run on thread-pool threads, while -`ProcessUpdates` reads/enumerates the same dictionaries each frame on the main thread -> -data race / mid-enumeration throw. **Fix:** marshal `Track` onto the main thread (queue it -like hot-swaps and apply in `ProcessUpdates`), or guard both dictionaries with a lock. -(Severity depends on the threading model — if `Load` is always awaited on the main thread -with a context installed, this drops to Low.) - ---- - -## Cluster D — Naming & public surface - -### F14 — Names that each mean two things *(Medium)* -- **`AssetJob`** is not a job — it's an already-computed result, but "Job" implies something - you `Start`/`await`. Rename -> `AssetResult` / `AssetOutcome`. -- **`On*`** is C#'s convention for *event handlers* (`OnClick`), yet here they're predicates - — and the same words are the `Match` callback parameters. Rename survivors to the - `Try*`/`Is*` convention. -- **"Cache"** names both the volatile in-memory store and the on-disk store; **"Load"** - names both the full orchestration (`AssetManager.Load`) and decode-one-file - (`AssetFileCache.Load`). Rename by role: `AssetStore` / `AssetDiskCache`, and give the - file method a narrower verb (`TryReadFromDisk`). - -### F15 — Public surface leaks the plumbing *(Medium)* -A user needs: register transcoders, `Load(id)`, scope for lifetime. But `Encode`, -`Decode` (returning `AssetJob`), and thereby the whole tri-state/EDI machinery are public -too. **Fix:** make `Encode`/`Decode`/`AssetJob` `internal` (with `InternalsVisibleTo` for -tests). Every paradigm hidden from users is a finding that stops being their problem. This -also lets F9's `AssetFileCache` fold into a private `AssetManager.TryReadFromDisk` — after -the trims above it's ~5 lines and holds nothing the manager can't reach. - -### F16 — `IAssetTranscoder` density *(Low)* -The two-arity settings-erasure is a legitimate, clever technique — but it mixes -accessibilities *inside* one interface (public `HotSwap` beside internal `Encode`) and -bridges via default-interface methods, so "how is a transcoder invoked?" needs two -interfaces read at once. **Fix:** move the erased members to an internal -`IAssetTranscoderCore`, and add one XML-doc line on it: "you implement -`IAssetTranscoder`; the erased base is bridged for you." - ---- - -## Suggested order of attack - -1. **F1, F2, F3, F4** — four small, mechanical fixes; correctness blockers (add two - `return`s, fix one variable, snapshot before removing). -2. **F11 + F6** — unify the tail and move to exhaustive consumption, so F1-F3 can't return. -3. **F8 + F7** — decide whether `Failure`/EDI stays; collapsing to nullable+throw erases - most of the "too many paradigms" feeling. -4. **F12, F13** — lifetime authority + `Track` threading (the hot-reload half of the worry). -5. **F14, F15, F9, F10, F16** — naming, surface, and the smaller trims. - -The three agents were unanimous that the tri-state result API is the load-bearing issue: it -caused the correctness bugs *and* it's the main source of the "multiple paradigms" feeling — -fix that cluster and most of the rest gets smaller on its own. diff --git a/Research/AssetPipelineRewriteReview.md b/Research/AssetPipelineRewriteReview.md deleted file mode 100644 index eb89218..0000000 --- a/Research/AssetPipelineRewriteReview.md +++ /dev/null @@ -1,726 +0,0 @@ -# Asset Pipeline Rewrite Review - -> Multi-agent review of the rewritten `source/CapriKit.AssetPipeline` and -> `source/CapriKit.AssetPipeline.DirectX11` (branch `feature/asset_pipeline`, commit `5a4a056` -> "Complete redo asset pipeline"). Four review agents ran in parallel — correctness, -> ease-of-use/DX/clarity, performance, and modern .NET constructs — each reading the full -> source plus its `CapriKit.IO` dependencies. Findings were deduplicated, re-ranked and -> cross-checked against a manual read; the blocking section was verified independently -> against the source before being written down. -> -> Supersedes `AssetPipelineReview.md`, which reviewed the *pre-rewrite* design. - -**Headline: the code cannot currently run end-to-end.** `AssetManager`'s constructor throws -for exactly the configuration `AddAssetPipeline` builds. That is consistent with the rest of -the evidence — commit `5a4a056` deleted all five `CapriKit.Tests/AssetPipeline/*` files without -replacement, and nothing in the repo calls `AssetManager.Load`. Read this document as a review -of a design that has not executed yet, not of working code. - -## Overview - -| # | Cluster | Finding | Severity | Location | -|---|---------|---------|----------|----------| -| B1 | Blocking | `AssetManager` ctor always throws — `Watch(DirectoryPath.Empty)` is rejected by every filesystem | Critical | `HotReloadManager.cs:38` | -| B2 | Blocking | Every shader with an `#include` fails to build — include root resolved against process CWD | Critical | `VertexShaderTranscoder.cs:14` | -| B3 | Blocking | Assets are never disposed — the `Line` wrapper is cast to `IDisposable`, not the asset | Critical | `AssetCache.cs:79,87` | -| B4 | Blocking | `AssetManager.Dispose` never disposes `HotReloadManager`, leaking the OS watcher | High | `AssetManager.cs:117` | -| C1 | Threading | Threading model undecided: async continuations mutate main-thread state | High | `AssetManager.cs:110`, `AssetCache.cs:62`, `HotReloadManager.cs:51` | -| C2 | Correctness | `#include`d files never hot-reload — dependency keys and watcher events use different path shapes | High | `VertexShaderTranscoder.cs:13-14` | -| C3 | Correctness | A failed encode destroys the last good build artifact | Medium | `AssetEncoder.cs:22` | -| C4 | Correctness | Concurrent `Load` of the same id throws | Medium | `AssetManager.cs:24` | -| C5 | Correctness | `IsUpToDate` never checks the primary source; `<` should be `!=` | Medium | `AssetManager.cs:93,102` | -| C6 | Correctness | Successful hot reload logged at `LogLevel.Error` | Low | `HotReloadManager.cs:161` | -| C7 | Correctness | Leftover `v2` namespace from the rewrite | Low | `HotReloadManager.cs:8` | -| D1 | DX | `Load` returns an `IDisposable` the caller must never dispose | High | `AssetManager.cs:24` | -| D2 | DX | The reader-lifetime rule is a `//` comment, so it never reaches IntelliSense | High | `IAssetTranscoder.cs:28-30` | -| D3 | DX | `AssetId(string Key, FilePath Path)` — swapping the arguments compiles | Medium | `Asset.cs:10` | -| D4 | DX | Build artifacts land beside sources; `.cka` not in `.gitignore` | Medium | `AssetUtilities.cs:11` | -| D5 | DX | Transcoder + settings passed on every single `Load` | Medium | `AssetManager.cs:24` | -| D6 | DX | Documentation gaps and one misleading `` | Medium | `IAssetTranscoder.cs:6-16` | -| D7 | DX | Error messages: one bare `Exception`, one raw `KeyNotFoundException`, one silent blackout | Medium | `AssetCache.cs:27,56`, `AssetDecoder.cs:78` | -| P1 | Perf | Every load reads the whole file twice | High | `AssetManager.cs:35,38` | -| P2 | Perf | `ArrayPool.Shared` silently stops pooling above 1 MB → LOH churn | High | `AssetDecoder.cs:24,60` | -| P3 | Perf | No `ConfigureAwait(false)` anywhere; a 50–500 ms `D3DCompile` can land on the main thread | Medium | 12 awaits across 4 files | -| P4 | Perf | First-run path throws an exception as normal control flow | Medium | `AssetDecoder.cs:55` | -| P5 | Perf | Redundant `stat` syscalls, per-load `Guid.Parse`, per-load transcoder allocation | Low | `AssetDecoder.cs:17,54` | -| M1 | Modern | Constrain `TSettings : IEquatable`; `NoSettings` as `readonly record struct` | High | `IAssetTranscoder.cs:17`, `AssetManager.cs:81-90` | -| M2 | Modern | No `CancellationToken` anywhere, unlike the rest of `CapriKit.IO` | Medium | `AssetManager.cs:24`, `IAssetTranscoder.cs:26` | -| M3 | Modern | Assorted one-line modernisations, no downside | Low | various | -| M4 | Modern | Library is AOT/trim-clean but unguarded — `IsAotCompatible` set nowhere | Low | `Directory.Build.props` | - -**Ground truth** (verified): `net10.0` (`net10.0-windows` for DirectX11), `LangVersion Latest` → C# 14, -`Nullable enable` with `WarningsAsErrors`, version `0.1.0-alpha`. No AOT/trim properties set anywhere -in the repo. Both projects compile with 0 warnings, so nothing below is flagged by the compiler. - ---- - -## Blocking — prevents the pipeline running at all - -### B1 · `AssetManager`'s constructor always throws - -`HotReloadManager.cs:38`: - -```csharp -Watcher = fileSystem.Watch(DirectoryPath.Empty); // Watch for all changed, usually fileSystem is a ScopedVirtualFileSystem -``` - -`DirectoryPath.Empty` is `new(null)` (`DirectoryPath.cs:12`), so `Path == ""` and -`IsAbsolute` is `Path.IsPathFullyQualified("")` → `false`. -`ReadOnlyScopedFileSystem.Watch` (`ScopedFileSystem.cs:143`) begins with -`ThrowIfPathIsOutsideBasePath(directory)`, which `Debug.Assert(path.IsAbsolute)` fires on in -Debug builds and then throws `ForbiddenPathException` because `""` does not start with the base -path. A plain `FileSystem` is no better: `Path.GetFullPath("")` throws `ArgumentException`. - -`AssetManager` constructs a `HotReloadManager` unconditionally (`AssetManager.cs:21`), and -`AddAssetPipeline` (`ServiceCollectionExtensions.cs:15`) supplies exactly a -`new FileSystem().ScopedTo(assetDirectory)`. So the documented entry point throws before you can -load anything. - -The intent is already stated in the comment on that line, and the overload exists: - -```csharp -Watcher = fileSystem.Watch(); // ScopedFileSystem.cs:151 — watches BasePath, no path check -``` - -It currently lives only on `ReadOnlyScopedFileSystem`, so it needs promoting to -`IReadOnlyVirtualFileSystem` to be reachable through the interface. The alternative is to make -`ThrowIfPathIsOutsideBasePath` treat an empty relative path as "the base path itself". - -### B2 · Every shader with an `#include` fails to build - -`VertexShaderTranscoder.cs:11-15`: - -```csharp -var source = await fileSystem.ReadAllText(id.Path); -var includePath = id.Path.Directory; // relative, e.g. "shaders/" -var bytes = ShaderCompiler.CompileVertexShader(fileSystem, includePath, source, id.Key, id.ToString()); -``` - -`includePath` is relative. `ShaderIncludeResolver`'s constructor wraps it in a -`ReadOnlyScopedFileSystem`, whose constructor does `BasePath = basePath.ToAbsolute()` -(`ScopedFileSystem.cs:49`) → `Path.GetFullPath("shaders/")`, resolved against -**`Environment.CurrentDirectory`**. - -With assets at `C:/game/content` and the process running from `C:/game/bin`, the resolver builds -`C:/game/bin/shaders/foo.hlsl` and hands it to the underlying scoped filesystem, which rejects it -with `ForbiddenPathException`. Shaders only compile when CWD happens to equal the asset root. - -Fix: resolve the include directory against the filesystem's base rather than the CWD — pass the -already-scoped filesystem plus the relative directory and let `ShaderIncludeResolver` keep paths -relative instead of calling `ToAbsolute()`. - -### B3 · Assets are never disposed - -`AssetCache.cs:76-80` and `:83-90`: - -```csharp -Lines.Remove(key, out var entry); -(entry as IDisposable)?.Dispose(); // entry is Line, not entry.asset -``` - -`Line` (`AssetCache.cs:11-15`) does not implement `IDisposable`, so the `as` cast is *always* null -and the `?.` is always a no-op. The same defect is in `Dispose()`, where `value` is also a `Line`. - -Load a shader, unload it, call `Update()`: the entry leaves the dictionary and the `VertexShader` -— along with its `ID3D11VertexShader` — is silently dropped. Every collected asset leaks its -native handle, `AssetManager.Dispose()` leaks all live ones, and D3D11 will report live-object -warnings on device release. - -```csharp -(entry?.asset as IDisposable)?.Dispose(); -(value.asset as IDisposable)?.Dispose(); -``` - -The naming inside `Line` is what hides this — PascalCase primary-constructor parameters shadowing -camelCase public fields, five lines apart. Renaming the type to `Entry` and the field to `Asset` -makes the bug visible at a glance. - -### B4 · `AssetManager.Dispose` never disposes `HotReloadManager` - -`AssetManager.cs:117-120` disposes only `Cache`. `HotReloadManager.Dispose` is the only caller of -`Watcher.Stop()` (`HotReloadManager.cs:148`), and grep confirms nothing in the repo invokes it. -After disposal the `FileSystemWatcher` keeps running, keeps enqueueing into -`FileSystemEventQueue`, and keeps the whole manager graph — including disposed D3D wrappers — -alive. Creating and disposing several `AssetManager`s leaks one OS watcher handle each. - ---- - -## Correctness - -### C1 · The threading model is undecided — and that is the root defect - -Three separate findings share one cause. `AssetManager.Load` is `async`; with no -`SynchronizationContext` (normal for a game loop) the continuation after -`await AssetDecoder.Decode(...)` runs on a thread-pool thread. `RegisterAsset` -(`AssetManager.cs:110-115`) then mutates main-thread state from there: - -- **`AssetCache.Collect`/`Dispose` take no lock** (`AssetCache.cs:62`, `:83`) while `Put`, `TryLease` - and `Return` all do. `Lines.Add` from a loader thread during `Collect`'s `foreach` gives - `InvalidOperationException: Collection was modified`, or a read racing a dictionary resize. -- **Zero-refcount resurrection.** `Collect` snapshots a `refCount == 0` entry into `toCollect` - (`:64-72`); before the removal loop at `:76` a background `TryLease` can bump it to 1 and hand the - asset to a caller. `Collect` then removes and (once B3 is fixed) disposes it. The caller holds a - disposed asset, and its later `Unload` throws `KeyNotFoundException` at `:56`. -- **`HotReloadManager` has no synchronisation at all.** `Track` (`:51-68`) writes the plain - `Dictionary` fields `Tracked` and `Dependents` while `DrainFileChanges` (`:91`) and `ReloadOne` - (`:114`) read them on the main thread. - -The class doc at `AssetCache.cs:6` also contradicts itself: "Assets can be leased and returned at -any time (though the class requires single-threaded access)" describes a class that nonetheless -takes a `Lock`. - -**Cheapest coherent fix:** have `RegisterAsset` enqueue onto a `ConcurrentQueue` that `Update()` -drains on the main thread, mirroring how `PendingReloads` already works. The cache then genuinely -*is* main-thread-only, the `Lock` can be deleted rather than extended, and the doc comment becomes -true. Decide this before touching anything else in `AssetCache` — it determines whether the lock -stays at all. - -### C2 · `#include`d files never hot-reload - -The transcoder reads its primary file straight off the spy -(`fileSystem.ReadAllText(id.Path)`), so the spy records a **relative** path. Includes go through -`new ReadOnlyScopedFileSystem(spy, includePath)`, whose `OpenRead` calls -`Source.OpenRead(GetFilePath(file))` — resolving to **absolute** before the spy ever sees it. -Observed in an agent's repro: - -``` -spy recorded: shaders/a.hlsl -spy recorded: C:/Users/.../capri_repro_assets/shaders/a.hlsl -``` - -Meanwhile `ScopedFileSystemEventListener.cs:22` normalises every watcher event to -`e.File.GetPathRelativeTo(BasePath)` — always relative. So `DrainFileChanges` -(`HotReloadManager.cs:91`) looks up a relative key in a `Dependents` map keyed absolutely, and -misses. - -Edit a top-level `.hlsl` and it hot-reloads; edit an `#include`d one and nothing happens, ever. -`IsUpToDate` still works for these files because `ScopedFileSystem.Exists` accepts both shapes — -which is precisely why this would be easy to ship unnoticed. - -Fix: normalise dependency keys to one representation. Simplest is to have -`VirtualFileSystemSpy` record `path.GetPathRelativeTo(basePath)`. - -### C3 · A failed encode destroys the last good build artifact - -`AssetEncoder.cs:22` opens the `.cka` with `FileMode.Create` (truncate) *before* -`encoder.Encode` runs. A shader typo during hot reload throws out of `WritePayload` and leaves a -0-byte `.cka`. It is recovered on the next run — `TryDecodeBuildMetaData`'s blanket `catch` -(`AssetDecoder.cs:78`) returns null and forces a rebuild — so this is not silent corruption, but -the previously-good artifact is gone. Encode into a buffer first and open the output file only -once encoding succeeded. This is also a prerequisite for cancellation being safe (see M2). - -### C4 · Concurrent `Load` of the same `AssetId` throws - -There is no in-flight de-duplication. Two overlapping calls both miss `TryLease`, both call -`AssetEncoder.Encode` on the same output path (the second `CreateReadWrite` throws `IOException`), -and if they get past that, the second `Cache.Put` throws -`new Exception($"Cache already contains asset: {id}.")` (`AssetCache.cs:27`). The same collision -exists between a `Load` and a concurrent `HotReloadable.Reload` of the same asset. -Fix: a `Dictionary>` of in-flight loads. - -### C5 · `IsUpToDate` never checks the primary source, and compares timestamps with `<` - -`AssetManager.cs:93-105` iterates only `build.Dependencies`, which is whatever the transcoder -happened to open *through the spy*. `AssetEncoder.cs:19`'s `ThrowOnFileNotFound(id.Path, fileSystem)` -uses the raw filesystem, and `Exists` is not spied, so nothing guarantees `id.Path` appears in the -list. A transcoder that generates content or reads its source by another route produces an empty -dependency list — the `foreach` body never runs, `IsUpToDate` returns `true` unconditionally, and -the stale artifact is used forever. Always check `id.Path` explicitly, and/or seed the spy with it. - -Separately, `:102` uses `if (version < lastWrite)`. Restoring an older copy of a source file with -its mtime preserved (backup restore, `robocopy`, some VCS tooling) leaves `lastWrite <= version`, -so no rebuild happens and the wrong asset is used. `version != lastWrite` is the standard -formulation. - -### C6 · Successful reload logged at `Error` - -`HotReloadManager.cs:161` — `[LoggerMessage(Level = LogLevel.Error, Message = "Reloading asset completed: {asset}")]`, -copy-pasted from the adjacent `LogReloadFailed`. Every successful hot reload is reported as an -error to the log sink and to anything filtering on error level. - -### C7 · Leftover `v2` namespace - -`HotReloadManager.cs:8` still declares `namespace CapriKit.AssetPipeline.v2;` while every other -file moved to `CapriKit.AssetPipeline`; `AssetManager.cs:1`'s `using CapriKit.AssetPipeline.v2;` -exists purely to compensate. `git show --stat 5a4a056` shows `v2/` → `` renames for `AssetCache` -and `HotReloadable` but not for this file. A `v2` sub-namespace is visible to anyone typing -`CapriKit.AssetPipeline.` in IntelliSense. - -### Checked and found clean - -Worth recording, so these are not re-reviewed later: - -- **Encode/decode round trip.** Every field traced. `AssetEncoder` writes - `Guid, int | int len, settings | int len, payload | int count, (long ticks, string path)*` and - `AssetDecoder` reads exactly that, in that order, matching the format comment at - `AssetEncoder.cs:9`. Endianness matches (`BinaryPrimitives` LE on both sides), `Guid` uses - `bigEndian: false` both ways, `Write(FilePath)` binds to the `string` overload via the implicit - conversion and pairs correctly with `ReadString`'s 7-bit-length prefix, and `SliceUnread` - correctly advances the outer reader past each length-delimited section. No mismatch found. -- `build != default` (`AssetManager.cs:36`) does behave as a null check — the record-synthesized - `op_Inequality` handles null correctly. (Still worth changing for clarity; see M3.) -- The watcher **is** debounced (`HotReloadManager.cs:78-82`, 0.5 s of quiet), and the `.cka` is - written on the raw filesystem *before* the spy is created (`AssetEncoder.cs:22` vs `:24`), so - build artifacts are not recorded as their own dependencies. No infinite rebuild loop. -- `IVertexShader.HotSwap` (`VertexShader.cs:12-19`) correctly preserves the live object's identity - and disposes only the orphaned old `ID3D11VertexShader`, so existing holders stay valid. -- `AssetDecoder`'s `ArrayPool` rent/return is balanced in `finally` and the sequence is bounded to - the real length rather than the rented length. - ---- - -## Ease of use, DX and clarity - -### D1 · `Load` hands back an `IDisposable` the caller must never dispose - -`AssetManager.cs:24` returns a bare `TAsset`. For the only worked example that is `IVertexShader`, -which is `IDisposable`. The C# reflex on an `IDisposable` returned from a method is `using` — -which destroys an asset the cache still hands to every other caller, while the refcount never -notices. `Unload(id)` is the real release, and nothing in the type system, the name, or the docs -says so: `Load` and `Unload` have no XML documentation at all. - -The failure is symmetric. Over-unloading is silent — `AssetCache.Return` (`:52-59`) decrements past -zero without complaint and `Collect` then frees a live asset. Forgetting to unload leaks with no -diagnostic. - -**Small fix** — document the ownership contract and make `Return` refuse to go negative: - -```csharp -/// -/// Loads an asset, building it first if there is no up-to-date build artifact. -/// The returned instance is owned by the AssetManager and shared with every other -/// caller of the same : never dispose it, and never keep it past the -/// matching . Each successful Load takes one reference; call -/// exactly once per Load. -/// -``` - -```csharp -// AssetCache.Return -if (!Lines.TryGetValue(id, out var entry)) - throw new InvalidOperationException($"Cannot unload asset {id}: it was never loaded, or it was already unloaded once per Load."); -if (--entry.refCount < 0) - throw new InvalidOperationException($"Asset {id} was unloaded more often than it was loaded."); -``` - -**Medium fix**, matching the `AssetRef` idea in `AssetPipelineArchitecture.md`: return an -`AssetLease : IDisposable` with a `.Value`, so `using` becomes the *correct* thing rather -than the destructive one. Costs one struct plus `.Value` at every use site. - -### D2 · The most dangerous rule in the contract is in a `//` comment - -`IAssetTranscoder.cs:28-30` states that the reader's buffer is only valid for the duration of the -call and decoders must copy out anything they keep. That is true — `AssetDecoder.cs:24,43` rents -from `ArrayPool` and returns it in a `finally` — and it is invisible to a package consumer, -because `//` comments do not ship in the XML documentation file or IntelliSense. A decoder that -retains a `ReadOnlySequence` slice compiles fine and later reads recycled pool memory under load. - -Promote it to `` on `Decode`, along with the async rationale at `:20`: - -```csharp -/// -/// The reader is backed by a pooled buffer that is recycled the moment this method returns: -/// copy out (ToArray, CopyTo) anything you keep. Storing the reader, a slice of -/// it, or a span into it will silently read another asset's bytes later. -/// Runs synchronously, possibly on a worker thread; do not touch main-thread-only state. -/// -``` - -### D3 · `AssetId(string Key, FilePath Path)` — swapping the arguments compiles - -`Asset.cs:10`. `FilePath` has `implicit operator FilePath(string?)` (`FilePath.cs:104`), so -`new AssetId("shaders/basic.hlsl", "VsMain")` — path first, the order every reader will guess — -compiles cleanly and fails at runtime with `FileNotFoundException: ... VsMain`. There is also no -way to express "no sub-resource" other than `new AssetId("", path)`. - -```csharp -/// Virtual path of the source file the asset is built from. -/// Names a sub-resource inside — e.g. the HLSL entry -/// point for a shader. Empty means "the whole file". Assets with the same Path but different -/// Keys are separate assets with separate build artifacts. -public sealed record AssetId(FilePath Path, string Key = ""); -``` - -There are zero call sites today, so this is free now and will not be later. - -### D4 · Build artifacts land beside the sources - -`AssetUtilities.cs:7-16` maps `grass.png` → `grass.png.cka` **in the same directory**, and `.cka` -is not in `.gitignore`. `AssetPipelineArchitecture.md` specifies a mirrored compiled tree, -deliberately separate. First-run experience is that the user's art folder fills with `.cka` files -and their next `git status` is noise. - -Small: add `*.cka` to `.gitignore` and hoist the extension to a documented -`public const string BuildArtifactExtension`. Real fix: a `DirectoryPath outputDirectory` on the -`AssetManager` constructor and `AddAssetPipeline`, with `ToEncodedFilePath` rebasing onto it -(~15 lines, and it is the design already written down). - -### D5 · Transcoder and settings on every `Load` - -`AssetManager.cs:24` requires two extra arguments per call, both constant for the lifetime of the -program. `AssetManagerExtensions.cs:9-13` shows what people reach for and why it is not enough: - -```csharp -public static Task LoadVertexShader(this AssetManager assetManager, Device device, AssetId id) -{ - var transcoder = new VertexShaderTranscoder(device); // new instance per call - return assetManager.Load(id, transcoder, default); -} -``` - -A fresh transcoder per call means `HotReloadManager.Track` stores a distinct transcoder object per -asset, and `device` is threaded through every call site forever. - -Recommended small fix — bind transcoder and settings once, keep full compile-time typing, add no -registry and no "missing transcoder" runtime failure: - -```csharp -/// How to build and load one kind of asset. Create once, reuse for every Load. -public sealed record AssetSource(IAssetTranscoder Transcoder, TSettings Settings) - where TAsset : class; - -public Task Load(AssetId id, AssetSource source) where TAsset : class - => Load(id, source.Transcoder, source.Settings); -``` - -Alternatives and their cost: *registration* (`assets.Register(...)` + `Load(id)`) -reads best at the call site but reintroduces the `TranscoderCollection` deleted in `5a4a056`, -trades a compile error for a runtime "no transcoder registered", and forces settings to be -per-type rather than per-load. *DI* is worse here, because the transcoder needs the `Device` and -the container would have to know about graphics. A typed `AssetHandle` bundling id + transcoder -+ settings is the nicest end state but wants a per-`Device` catalog object. - -### D6 · Documentation - -Measured against the standard in `CLAUDE.md` ("teach the library user how to use this correctly -and when not to"). - -**Undocumented public members:** `AssetManager` itself — the entry point of the package — plus its -constructor, `Load`, `Unload` and `Dispose`; `IAssetTranscoder.Id` and `.Version`; `NoSettings`; -all four abstract members of `NoSettingsTranscoder`; `AddAssetPipeline`; -`AssetManagerExtensions.LoadVertexShader`. - -`Id`/`Version` is the highest-value gap in the file: they are the cache-invalidation keys, and -nothing tells an implementer that `Version` must be bumped whenever `Encode`'s output format -changes — otherwise every user's stale artifacts are fed to the new decoder. - -**Misleading docs:** - -- `IAssetTranscoder.cs:6-9` — "Interface for classes that builds assets ... and load them when the - program needs them" describes `AssetManager`, not a transcoder. `:16`'s - `` then copies that wrong sentence onto the generic - interface people actually implement. Better: "One transcoder owns both halves of a single asset - type: the offline build (`Encode`, dev machine) and the runtime load (`Decode`, game process)." -- `:33` — "Decodes the file created the `Encode`" is missing a word. -- `:39-50` — `WriteSettings` and `ReadSettings` have near-identical text, and `ReadSettings` says - it "Decodes the settings ... **into** the stream" when it reads *from*. Neither mentions that - `WriteSettings` doubles as the staleness comparison (see M1). -- `AssetCache.cs:6` — claims single-threaded access while the class locks. The truth is narrower - and more useful: lease/return are thread-safe; `Collect` and `Dispose` are main-thread only. -- `AssetManager.cs:64` — `Update` has a `` but no ``, so IntelliSense shows an - empty description with a floating remark. It never says what `Update` does or what breaks if you - skip it (nothing is ever freed; hot reload silently does nothing). Consider a - `[Conditional("DEBUG")]` main-thread assertion so "must" is enforced rather than hoped for. - -### D7 · Failure modes surfaced to the user - -| Situation | Today | Verdict | -|---|---|---| -| Source file missing | `FileNotFoundException` naming the path — `AssetManager.cs:48` | Good | -| Build artifact missing at decode | `FileNotFoundException` naming file and asset — `AssetDecoder.cs:19` | Good | -| Artifact built by another transcoder | `InvalidDataException` naming both GUIDs and versions — `AssetDecoder.cs:132` | Good | -| `ThrowOnFileNotFound` in the encoder | `new FileNotFoundException(null, path)` — `AssetUtilities.cs:22` | Bad: `null` message, no asset context | -| Same id loaded twice concurrently | `throw new Exception(...)` — `AssetCache.cs:27` | Bad: uncatchable by type | -| `Unload` of an id never loaded | `KeyNotFoundException` — `AssetCache.cs:56` | Worst: names neither the asset nor the API | -| Same id loaded as two different types | `InvalidCastException`, no id — `AssetCache.cs:44` | Bad | -| **Corrupt build artifact** | `catch { return null; }` — `AssetDecoder.cs:78` | Worst for diagnosis | - -That last row is the answer to "what does it say when an artifact is corrupt": *nothing*. It -silently rebuilds, which is the right recovery, but a permission error, a >2 GB -`OverflowException` (`AssetDecoder.cs:59`) and a genuine bug in someone's `ReadSettings` all look -identical and produce a rebuild on **every single load**, forever, with zero log output — -`TryDecodeBuildMetaData` has no `ILogger`. Pass the logger in and log once at Debug/Warning. - -### Naming - -- `AssetCache.Line` — "line" is a CPU-cache term; this is an entry, and the local variable is - literally called `entry`. Renaming to `Entry` (and the field to `Asset`) is what exposes B3. -- `Asset` — a type named `Asset`, with a type parameter named `TAsset`, and a - member `Value` of that type. `LoadedAsset` reads better; it is internal, so this is cheap. -- `HotSwap(TAsset instance, TAsset newParts)` — `newParts` is not parts, it is a fully constructed - replacement. `HotSwap(TAsset live, TAsset replacement)`, and the doc should state who disposes - the leftover wrapper: `VertexShader.cs:12-19` disposes the old `ID3D11VertexShader` but not the - `newParts` wrapper, and that asymmetry needs saying. -- **"Transcoder" already means something else in this repo.** `CapriKit.SuperCompressed` has - `Ktx2Transcoder`, using *transcode* in its Basis-Universal sense: compressed container → GPU - format, at runtime. A future `TextureTranscoder : IAssetTranscoder` - will call `Ktx2Transcoder.Transcode(...)` inside its `Decode` — two unrelated meanings of the - same word in one file. `AssetPipelineArchitecture.md` names these **Compilers** and **Loaders**. - The one-interface-does-both design is genuinely better than MonoGame's four-piece split and is - worth keeping; only the name collides. Suggest taking the doc fix now and revisiting the name - when the texture transcoder lands and the collision becomes concrete. - -### Ceremony cost in DirectX11 — proportionate - -| File | Lines | Real statements | -|---|---|---| -| `VertexShaderTranscoder.cs` | 29 | 6 | -| `ShaderTranscoder.cs` | 26 | 8 | -| `AssetManagerExtensions.cs` | 14 | 2 | -| **Total per asset type** | **69** | **16** | - -`NoSettingsTranscoder` is doing its job — roughly 4:1 lines-to-logic for a binary-serialized GPU -resource is fine, and there is no per-type "content writer + content reader + settings class" tax -like MonoGame's. The only remaining pure forwarding is `HotSwap` (one line delegating to the -asset's own `HotSwap`); an optional `HotSwappableTranscoder` base would remove 4 lines per -transcoder but is not worth it until there are three or more. - ---- - -## Performance - -Judged against two standards: the **encode** path runs rarely and offline-ish, so allocation -matters little; the **load** path runs during gameplay or level load, where allocations, copies -and main-thread stalls are expensive. `Update()` runs on the main thread every frame. - -### P1 · Every load reads the whole file twice - -`AssetManager.cs:35` calls `TryDecodeBuildMetaData`, which `ReadExactlyAsync`s the *entire* file — -payload included — just to `SkipPayload` (`AssetDecoder.cs:69`) and reach the dependency list at -the tail. Then `:38` calls `Decode`, which re-opens and re-reads the identical bytes. - -Cost: 2× file bytes, 2× `FileStream` open, 2× buffer rent, per asset, on the common path. A 4 MB -texture costs 8 MB of IO. File open on Windows is ~50–500 µs cold, so the duplicate opens alone -are ~50–500 ms per 1000 assets. - -Fix: one pass. Read the file once into a pooled buffer, parse header → settings → *retain the -payload slice* → skip to dependencies → run the up-to-date check → and only then call -`transcoder.Decode` on the retained slice. `SliceUnread` already gives exactly the slice needed; -the file format does not change. **Biggest single item in this review.** - -### P2 · `ArrayPool.Shared` silently stops pooling above 1 MB - -`AssetDecoder.cs:24,60`. `ArrayPool.Shared` caps at `MaxArrayLength = 1024*1024`; above that -`Rent` calls `GC.AllocateUninitializedArray` and `Return` drops the array on the floor. So for any -asset over 1 MB — every texture and mesh — this is a plain **LOH allocation of the full file size, -twice per load** (see P1), each becoming garbage immediately. A 16 MB texture is 32 MB of LOH -garbage per load; 100 such assets is 3.2 GB of churn and forced gen2/LOH pressure during level -load, exactly the main-thread stall case that matters. - -Fix: a dedicated `ArrayPool.Create(maxArrayLength: 64MB, maxArraysPerBucket: 2)` held static -on `AssetDecoder`, or a single reusable staging buffer on `AssetManager` (loads are already -serialised through the same manager). Fixing P1 first halves this for free. - -### P3 · No `ConfigureAwait(false)` anywhere - -12 awaits across `AssetManager.cs`, `AssetDecoder.cs`, `AssetEncoder.cs`, `HotReloadable.cs`. -`CapriKit.IO` uses it on all 6 of its awaits, so the pipeline is inconsistent with its own -dependency. If a host installs a `SynchronizationContext` — an editor or tooling host, an -ImGui-driven tool, a test harness — every continuation posts back to the main thread. The worst -case is concrete: `VertexShaderTranscoder.cs:13-15` awaits `ReadAllText` and then runs a -synchronous `D3DCompile` of **50–500 ms** on the continuation. - -Add `.ConfigureAwait(false)` throughout and enable CA2007 so it stays enforced. Note this makes -C1's race explicit rather than worse — settle the threading model first, and do not treat adding -it as closing the threading question. - -Related: `Load` runs synchronously on the caller's thread up to the first real yield, including -the blocking `FileStream` open. Either document that `Load` must be called off the main thread, or -open the file on a worker. - -### P4 · The first-run path throws as normal control flow - -`AssetDecoder.cs:55` throws `FileNotFoundException` when the artifact does not exist yet, caught by -the blanket handler at `:78`. "Asset has never been built" is the expected first-run state, not an -error. Undebugged that is ~5–50 µs; **under a debugger, first-chance exception notifications cost -1–10 ms each**, so a fresh build of 1000 assets becomes 1–10 seconds of pure debugger stall and -floods the exception window. One line: `if (!fileSystem.Exists(inputPath)) { return null; }` before -the `try`. Cheapest high-value fix in this section. - -### P5 · Smaller items - -- **~6 redundant `stat` syscalls per load.** `AssetDecoder.cs:17,54` call `Exists`, then - `OpenRead` → `FindOrThrow` → `GetFileInfo().Exists`, then the actual open — and all of it twice - because of P1. `FileMode.Open` already throws `FileNotFoundException`, so `FindOrThrow` is pure - duplication. ~30–90 ms per 1000-asset level load; the fix is *fewer* lines of code. -- **`Guid.Parse` on every transcoder construction** (`VertexShaderTranscoder.cs:9`) combined with a - new transcoder per `Load` (`AssetManagerExtensions.cs:11`). `static readonly Guid` and a cached - per-`Device` transcoder. -- **`IsUpToDate`'s two `ArrayBufferWriter`s** are ~64 bytes per load for `NoSettings` and ~600 - bytes for a realistic settings struct — real but noise next to P1/P2. The *structural* waste - matters more: `build.Settings` was just deserialised purely so it could be re-serialised and - compared byte-for-byte. See M1. -- **`ArrayBufferWriter` growth in `AssetEncoder`** (`:53`) doubles and discards, never using - `ArrayPool`; combined with `PipeWriter` buffering everything until `FlushAsync` at `:31`, peak - memory is ~3× payload size. Normally an encode-path shrug, except this runs during hot reload - while the game is live, so the LOH churn causes gen2 pauses mid-play. Dropping `PipeWriter` for a - single size-hinted `ArrayBufferWriter` + `output.WriteAsync` is simpler *and* faster. - -### Explicitly judged not worth it - -- **`Collect()`'s O(n) scan.** Genuinely zero-allocation in steady state (struct enumerator; - `toCollect` only allocates when there is something to collect). ~5–20 µs/frame at 1000 assets - ≈ 0.1% of a 16.6 ms budget. Restructuring to a candidate list pushed from `Return()` is worth - doing for *clarity*, and the O() improvement is a bonus — not the reason. -- **`ValueTask` for `Load`.** ~72 bytes per load against a public signature change and - `.AsTask()` at any call site that stores the result. If synchronous cache hits matter, the honest - fix is a genuinely synchronous `TryGetLoaded(id, out asset)`, which needs no `ValueTask`. -- **Lock contention in `AssetCache`.** Uncontended `Lock` is ~20 ns and `Load` is not per-frame. - Do not remove the lock for speed (C1 may remove it for a better reason). -- **`Stopwatch.GetElapsedTime` every frame** (`HotReloadManager.cs:78`) — ~20–25 ns. Negligible. - -### Benchmarks - -`CapriKit.Benchmarks` uses `BenchmarkSwitcher.FromAssembly`, so a `[MemoryDiagnoser]` class drops -straight in — but it currently references only `CapriKit.Concurrency` and `CapriKit.IO`, so a -project reference is needed. Worth measuring, against `InMemoryFileSystem` to isolate CPU from -disk: `Load` on the cold-but-up-to-date path (the direct before/after for P1 and P5), -`AssetCache.Collect()` vs entry count at 100/1,000/10,000 (the one estimate above derived from -first principles rather than measurement), and `AssetEncoder.Encode` peak allocation across -payload sizes. **Fix B1 first** — none of this is currently exercisable end-to-end. - ---- - -## Modern .NET - -Targeting `net10.0` / C# 14. House style already in use elsewhere in the repo, so these are safe: -`SearchValues` (`IOUtilities.cs:11-12`), `ValueTask` and optional `CancellationToken` on every -async IO helper (`VirtualFileSystemExtensions.cs:18-63`), `readonly record struct`, primary -constructors, collection expressions, `[LoggerMessage]`. Never used anywhere in the repo: -static abstract interface members, `FrozenDictionary`, `TimeProvider`, `CollectionsMarshal`, -generic math. `System.Threading.Lock` appears exactly once — in `AssetCache.cs:17`, so the code -under review is already the most modern locking in the repo. - -### M1 · `where TSettings : IEquatable` + `readonly record struct NoSettings` - -`AssetManager.cs:81-90` decides staleness by serializing *both* settings objects into fresh -`ArrayBufferWriter`s and comparing spans. Nothing in `IAssetTranscoder.cs:39-43` tells the -implementer that their serializer must be **deterministic** — iterate a `Dictionary`, write a -`float` through a culture-sensitive path, or include a timestamp, and every `Load` decides the -artifact is stale and rebuilds forever, with no error. - -**The agents disagreed here, and the disagreement is instructive.** One proposed -`EqualityComparer.Default.Equals(...)`; another argued *against* it, because it silently -degrades to reference equality for a settings type that forgets to implement equality — a -"rebuild forever" cliff that fails silently, i.e. exactly the bug it was meant to remove. Adding -the constraint answers that objection directly: - -```csharp -public interface IAssetTranscoder - where TAsset : class - where TSettings : IEquatable - -// AssetManager.IsUpToDate -if (!settings.Equals(build.Settings)) { return false; } -``` - -and `public readonly record struct NoSettings;` gets `IEquatable`, `Equals`, -`GetHashCode` and `==` from one word. Nine lines become one, two allocations and two serializations -leave every load, and the hidden determinism contract disappears rather than needing to be -documented. Source-breaking — which at `0.1.0-alpha`, with `NoSettings` as the only settings type -in the repo, is the right moment. - -### M2 · `CancellationToken` — a real gap, with a caveat - -No method in the pipeline takes a token, though `CapriKit.IO`'s async helpers all accept one and -`AssetDecoder.cs:28,64` calls `ReadExactlyAsync` without one. Add -`CancellationToken token = default` as the trailing parameter and thread it into -`IAssetTranscoder.Encode`, which is where the seconds actually go. - -Honest caveat: this buys nothing on its own. `ShaderCompiler.CompileVertexShader` is a synchronous -blocking call inside an `async` method, so a token only takes effect at the cheap IO awaits — and -cancelling mid-`Encode` leaves a truncated `.cka` (C3). Add the parameter, but treat the -"cancel the loading screen" story as unfinished until transcoders poll and the write is atomic. -Source-breaking for `IAssetTranscoder.Encode`, not for `AssetManager.Load`. - -### M3 · One-line modernisations with no downside - -- `build is not null` instead of `build != default` (`AssetManager.cs:36`) — `!= default` on a - record dispatches through the synthesized `op_Inequality` instead of a plain null test. -- `catch (Exception ex) when (ex is not OperationCanceledException)` at `AssetDecoder.cs:78` — the - bare `catch` today also eats `OutOfMemoryException`, and once M2 lands it would convert "the user - cancelled" into "the metadata was unreadable, rebuild from scratch". -- `toCollect ??= []` (`AssetCache.cs:69`). -- Seal the records (`Asset.cs:10,12,15`) — `AssetId` is a `Dictionary` key in three places, so - sealing removes the `EqualityContract` virtual call from every lookup and makes the classic - derived-record-never-equals-base cache miss impossible. `record class` is also a redundant - spelling of `record`. -- `[LoggerMessage]`: drop `static` and the `ILogger` parameter (the generator resolves the field on - the containing type; both classes have one), removing a repeated argument from 13 call sites. - PascalCase the placeholders per CA1727. And `AssetManager.cs:122-128` logs *every* asset load at - `Information` — `Debug` fits a level load of a few thousand assets better. -- `static readonly Guid` instead of `Guid.Parse` per construction - (`VertexShaderTranscoder.cs:9`). -- `PendingRebuilds.First()` (`HotReloadManager.cs:107`) boxes `HashSet.Enumerator` through - `IEnumerable`. A `Queue` alongside the `HashSet` gives dedup *and* deterministic - reload order instead of hash order. -- `ObjectDisposedException.ThrowIf(disposed, this)` on `Load`/`Update`/`Put`/`TryLease` — house - style already in `CapriKit.SuperCompressed`. Today, using an `AssetManager` after `Dispose` - silently succeeds against an emptied cache. -- Drop the redundant `public` on `IAssetTranscoder.cs:26,36,43,50` (`:59` already omits it). -- `FileChances` → `FileChanges` (`HotReloadManager.cs:22,39,89`). - -### M4 · AOT/trim: clean today, unguarded tomorrow - -No reflection, no `Activator.CreateInstance`, no `MakeGenericType`, no `dynamic`, no generic -*virtual* methods; `ServiceCollectionExtensions.cs:12` uses the lambda-factory `AddSingleton` -overload rather than reflection-based activation. **The library is AOT/trim-clean as written.** -`true` is set nowhere in the repo — adding it turns on the -analysers so the property stays true when a future transcoder reaches for reflection. One line, no -code change, and it is the kind of guarantee a game library gets asked for. - -### Considered and rejected - -- **`System.IO.Hashing` / `XxHash3` for freshness.** Content hashing means *reading every - dependency file* on every load — strictly more IO on the path you most want fast. The one place - it would pay is hot reload: an editor that rewrites a file unchanged currently triggers a full - rebuild, and hashing only the file that raised the event would skip that. Worth doing *if* - spurious rebuilds annoy you in practice. Note `System.IO.Hashing` is declared at - `Directory.Packages.props:20` but referenced by no project — a dead central-package entry. -- **`FrozenDictionary`.** Every map here is write-often (`Lines`, `Tracked`, `Dependents`). Frozen - collections are for build-once-read-forever. No fit. -- **Static abstract interface members for `Id`/`Version`.** Would make version drift a - compile-time fact, but transcoders are *instances* carrying state (`VertexShaderTranscoder` holds - a `Device`), so every method in `AssetEncoder`/`AssetDecoder`/`AssetManager` would need an extra - `TTranscoder` type parameter — four files churned for harder-to-read generic signatures. -- **`TimeProvider`** for the hot-reload debounce (`HotReloadManager.cs:17,47,78`) — a 1:1 API swap - that would make the 0.5 s `MinWaitTime` testable without `Thread.Sleep`. Take it only if you - intend to test the debounce; it needs `Microsoft.Extensions.TimeProvider.Testing` added to - `Directory.Packages.props` and the repo uses `TimeProvider` nowhere. -- **`MemoryPool` + `using`** instead of the `ArrayPool` `try/finally` — `SequenceReaders.Create` - takes a `byte[]`, so you would need `MemoryMarshal.TryGetArray` to get back out. Worse than what - it replaces. -- **`IAsyncEnumerable`, `field`, `required`/`init`, generic math.** No batch-load API to convert, - no hand-written backing fields, no object-initializer surface, no numeric-generic code. - ---- - -## Tests - -Commit `5a4a056` deleted `AssetDecoderTests.cs`, `AssetEncoderTests.cs`, `AssetManagerTests.cs`, -`DummyTranscoder.cs` and `RepeatTranscoder.cs`, and nothing replaced them — there is no -`AssetPipeline` folder under `CapriKit.Tests` at all, though `CapriKit.Tests.csproj` still -project-references `CapriKit.AssetPipeline` and `Directory.Build.props:103` already grants -`InternalsVisibleTo`. - -Per `source/CapriKit.Tests/README.md` the bar is the happy path on anything with some complexity. -A round-trip test — encode → decode → assert equal, plus a second `Load` that hits the cache — -against `InMemoryFileSystem` would be roughly 40 lines and would have caught B3 immediately. - -`CapriKit.AssetPipeline` is also one of only two `source/` projects without a `README.md`, and it -is the one whose file format (`AssetEncoder.cs:9`) and hot-reload threading model most need a page -of prose. - ---- - -## Suggested order - -1. **B1 and B2** — nothing runs without them. -2. **Write back the happy-path round-trip test.** It is the thing that turns the rest of this list - from review comments into a safety net. -3. **B3, B4** — small, mechanical, and each one is a resource leak. -4. **Decide the threading model (C1)** before touching `AssetCache` further; it determines whether - the `Lock` stays at all, and P3 depends on the answer. -5. **D3, D4, M1** while there are still zero call sites and the breaking changes are free. -6. **P1 and P4** — the two largest performance wins, and P1 subsumes several smaller items. -7. The documentation pass (D1, D2, D6) and the error-message cleanup (D7). diff --git a/Research/AssetPipelineVNextReview.html b/Research/AssetPipelineVNextReview.html deleted file mode 100644 index 5193e52..0000000 --- a/Research/AssetPipelineVNextReview.html +++ /dev/null @@ -1,675 +0,0 @@ - - - - - -Asset Pipeline vNext Review - - - - -
-
-
CapriKit · feature/asset_pipeline · c9c853b
-

Asset Pipeline vNext Review

-

Four independent reviews — correctness, thread-safety, API ergonomics, implementation size — of - source/CapriKit.AssetPipeline and its DirectX11 consumer. Findings below are merged by defect, so one entry may - carry badges from several reviews that hit it independently.

-
- Reviewed 1,195 lines / 13 files + 3 DX11 files - Entry point AssetManagerTests.LoadAsset - Date 2026-08-14 - Repros executed 6 of 8 critical/high claims -
- -
-
Solution build
FAIL
DX11 project, CS1061
-
Critical
4
3 with executed repros
-
High
6
5 in hot reload
-
Medium / Low
14
triage after the above
-
Hot reload
0%
never ran end‑to‑end
-
Removable
47%
1,195 → ~630 lines
-
-
-
- -
-
-

The headline

-

The passing test and the three critical bugs are not in tension — the test only ever exercises the - first-run path, and every critical defect lives on a path it cannot reach.

- -

[Before(Test)] creates a fresh temporary directory for each test, so no .cka build artifact ever - exists when the test runs. That means TryDecodeBuildMetaData always returns null, the up-to-date - branch is never taken, and the missing return that corrupts it is invisible. The test also loads exactly - one asset, of type string — which is not IDisposable, so the disposal bug cannot - manifest — and it never calls Dispose on the manager, which would throw. One green test, four criticals, - no contradiction.

- -
- What this means for the design -

None of the four criticals is a flaw in the vNext design. The promise/bundle/pull model from - AssetPipelineLoadingGroupsV2.md holds up; two of the criticals are the design's own written requirements - (“N−1 extra leases”, “faulted counts as resolved”) simply not implemented yet, and one is a - missing return statement. The architecture is sound. The wiring is not finished.

-
-
-
- -
-
-

Blockers

-

Fix these four before anything else. Nothing downstream can be trusted while they stand.

- -
-
B1The solution does not build — the only real consumer calls a two-iteration-old API
-
CapriKit.AssetPipeline.DirectX11/AssetManagerExtensions.cs:12
-
verifiedcorrectnessthreadingapisize
- 
error CS1061: 'AssetManager' does not contain a definition for 'Load'
-               and no accessible extension method 'Load' accepting a first
-               argument of type 'AssetManager' could be found
-

LoadVertexShader still calls Load(id, transcoder, default) returning Task<IVertexShader>. - The current Load is internal and takes (AssetId, TSettings). Because only - CapriKit.Tests has InternalsVisibleTo, the entire D3D11 side of the pipeline is unreachable from - outside the assembly — which is precisely why none of the races below have bitten a real renderer yet.

-

dotnet build CapriKit.AssetPipeline.csproj succeeds on its own, so this is invisible unless you build the - solution. Downstream convenience extensions can no longer hang off AssetManager at all; the public entry - point is now AssetBundleBuilder.

- Fix - 
public static AssetHandle<IVertexShader> LoadVertexShader(this AssetBundleBuilder builder, AssetId id)
-    => builder.Load<IVertexShader, NoSettings>(id, default);
-

The per-call new VertexShaderTranscoder(device) must also move to a one-time registration — - RegisterTranscoder throws on the second call for the same asset type.

-
- -
-
B2Missing return: every asset is rebuilt on every boot, then crashes Update()
-
AssetManager.cs:106-122 — no return after the up-to-date branch at :111
-
repro executedcorrectnessthreadingsize
-

The if (build != default && IsUpToDate(...)) block writes to Incoming and then - falls through into the rebuild path, so a current asset is decoded, then re-encoded and decoded again, - and published to the channel twice.

- Failure scenario — executed -

Build Hello.txt once so Hello.txt.cka exists. Restart with a fresh AssetManager over - the same directory. Update() drains message 1, resolves the handles, does - Outstanding.Remove(id). It then drains message 2 and hits Outstanding[id]:

- 
System.Collections.Generic.KeyNotFoundException: The given key
-'AssetId { Key = , Path = Hello.txt }' was not present in the dictionary.
-   at CapriKit.AssetPipeline.AssetManager.Update() in AssetManager.cs:line 145
-

Because the staleness fast path has therefore never once executed, the whole envelope and staleness subsystem is - far less load-bearing than its line count suggests — which lowers the risk of restructuring it (see S2).

- Fix — both halves - 
    LogLoadedFromFile(Logger, id);
-    return;                                  // <-- missing
-
-// and harden the drain so a stray message can never kill the frame loop:
-if (!Outstanding.Remove(id, out var handles)) { continue; }
-
- -
-
B3Two handles for one asset id hand out an already-disposed instance
-
AssetManager.cs:145-151 (materializer called per handle) · AssetCache.cs:36-43
-
repro executedcorrectnessthreading
- 
var handles = Outstanding[id];
-foreach (var handle in handles)
-{
-    var asset = materializer();   // <-- called once PER HANDLE
-    handle.Resolve(asset);
-}
-

The materializer closes over a single decoded instance. On the second call PutOrLease finds the entry already - present and enqueues new Entry(id, candidate, 0) for disposal — but candidate - is entry.Asset, the live object. Cache.Collect(), three lines later in the same frame, - disposes it. Both handles now hold a disposed asset that is still in Entries with refcount 2.

- Failure scenario — executed -

Two handles for one id: same instance: True, A.IsDisposed: True, B.IsDisposed: True. Under load — - 8 threads × 100 loads over 8 ids with a pumping Update() — - 585 of 800 bundles received a disposed asset, with zero exceptions raised. Nothing signals the corruption. - For a VertexShaderTranscoder this is a released ID3D11VertexShader handed to gameplay code.

-

The author's own ExampleAssets sample (AssetBundleLoader.cs:71-72) requests the same - AssetId twice, so the shipped usage sketch triggers this directly.

- Fix — this is the “N−1 extra leases” note from the V2 design - 
if (!Outstanding.Remove(id, out var handles)) { continue; }
-var asset = materializer();                  // exactly once
-Cache.AddLeases(id, handles.Count - 1);      // new: refcount += n under Lock
-foreach (var handle in handles) { handle.Resolve(asset); }
-

Additionally guard PutOrLease with ReferenceEquals(entry.Asset, candidate) and skip the enqueue. - Queueing a live object for disposal should be impossible, not merely unreachable.

-
- -
-
B4One failed load permanently kills Update() for the rest of the process
-
AssetManager.cs:83-88 · LightweightChannel.cs:48 · AssetManager.cs:140-153
-
repro executedcorrectnessthreading
-

LightweightChannel.Error is set once and never cleared; TryRead rethrows it - whenever the queue is empty. The worker's failure path writes there, so Update() throws on every subsequent - frame — from inside lock (RequestLock), before Cache.Collect() and - HotReloadManager.Update() are ever reached.

- Failure scenario — executed -

Request one missing file, then pump: Update() threw 199 of 200 times. Pending disposals - accumulate forever, no other asset can ever complete, hot reload stops, and the failed id stays in - Outstanding so its handles never resolve — IsReady returns false forever - rather than reporting the error.

- Fix -

Reserve the channel's sticky error slot for “the pump itself is broken”, and give the handle the faulted state - the V2 design already specified (“faulted counts as resolved”):

- 
internal ExceptionDispatchInfo? Error;
-internal bool IsResolved => Volatile.Read(ref value) is not null || Error is not null;
-

Route the failure per-id so it surfaces from its bundle's IsReady/Wait, once, instead of - poisoning the shared pump.

-
-
-
- -
-
-

Hot reload has never worked

-

Five separate defects compound here. The subsystem is the largest in the pipeline (287 lines, 24%) and the - least tested — its only test constructs the class and asserts nothing.

- -
-
H1Every reload throws: the encoder disposes a stream it does not own
-
AssetEncoder.cs:24 · symptom at HotReloadable.cs:39-43 · same bug in AssetDecoder.cs:24
-
repro executedcorrectness
-

using var output = outputStreamOverride ?? fileSystem.CreateReadWrite(outputPath) takes ownership of a caller's - stream. PipeWriter.Create(stream) defaults to LeaveOpen = false, so - CompleteAsync() already disposed it; the using disposes it again.

- 
REPRO4 caught: ObjectDisposedException: Cannot access a closed Stream.
-   at System.IO.MemoryStream.Seek(Int64, SeekOrigin)
-

So HotReloadable.Reload throws at stream.Seek(0, SeekOrigin.Begin) on every hot - reload. Combined with H2 below, hot reload has never completed once.

- Fix - 
var writer = PipeWriter.Create(output, new StreamPipeWriterOptions(leaveOpen: true));
-try { /* snip: write */ }
-finally { if (outputStreamOverride is null) { output.Dispose(); } }
-
- -
-
H2isReloading latches true forever, and is read across threads without a barrier
-
HotReloadManager.cs:32, 98, 146, 175-185
-
correctnessthreading
-

isReloading = true is set unconditionally at :146, but is only cleared inside the - FireAndForget callbacks attached to target?.Reload(...). When target is - null — the case the code's own comment calls normal — the null-conditional - short-circuits and nothing is ever attached. Update() then returns early on every subsequent frame: - no file-change draining, no reloads, and HotSwapPending() never runs again either.

-

Separately, the flag is written from a thread-pool continuation and read on the main thread as a plain - bool — a data race the JIT may resolve by keeping the read in a register.

- Fix - 
private volatile bool isReloading;
-// ...
-if (target is null) { isReloading = false; return; }   // before dispatching
-
- -
-
H3Hot reload can HotSwap an asset that Collect already disposed
-
HotReloadable.cs:19,31,35 · AssetCache.cs:88-92
-
threading
-

WeakReference.IsAlive answers “the GC has not collected it”, not “it has not been - disposed”. Return moves an entry to PendingDispose (still strongly reachable); - Collect disposes it; the object then stays GC-alive for an arbitrary time. ReloadOne sees - IsAlive == true, picks it as target, and HotSwap runs against a released COM object.

- Fix -

Liveness must mean “the cache still holds a lease”. Have the cache call - HotReloadManager.Untrack(id) on eviction, so IsAlive never depends on GC timing.

-
- -
-
H4Tracking dictionaries are read outside the lock that guards their writes
-
HotReloadManager.cs:155-171 (Tracked) · :122-130 (Dependents)
-
threading
- 
lock (TrackingLock) { candidates = Tracked[id]; }   // copies the REFERENCE only
-// ...
-foreach (var candidate in candidates) { ... }        // iterated unlocked
-

The lock covers the dictionary lookup and nothing else, while Track mutates that very list under the lock. - DrainFileChanges reads Dependents with no lock at all — and its XML doc - (“can only be used by one thread at a time”) directly contradicts Track's doc - (“thread-safe, multiple threads can call this method”).

-

Latent, not live. Track is currently only reachable from Update(), so - everything runs on the main thread today. It breaks the moment materialization moves off it — and the documentation - already promises the guarantee the code does not provide. Resolve the contradiction in one direction and delete - the // TODO: ensure that HotReloadManager.Track is thread safe comments either way.

-
- -
-
H5The dedupe guard compares two different types, so it is always false
-
HotReloadManager.cs:68
-
correctnessthreading
- 
if (assets.Any(a => object.ReferenceEquals(a, asset))) { return; }
-//                  ^ HotReloadable        ^ Asset<TAsset,TSettings>
-

Different types by construction, never reference-equal. It compiles, and the documented guarantee - “registering the same asset multiple times is safe” does not hold. Compounding it, - ReloadOne builds a pruned list at :162-170 and then throws it - away — so Tracked grows monotonically for the process lifetime, and dead ids keep feeding - PendingRebuilds, which is exactly the input that triggers H2's permanent latch.

-
-
-
- -
-
-

Remaining findings

-
- - - - - - - - - - - - - - - - - - -
IDSeverityFindingLocation
M1HighAssetManager.Dispose always throws (nothing ever calls Cache.Return), so HotReloadManager.Dispose never runs and the OS FileSystemWatcher leaks on every shutdownAssetManager.cs:159
M2MediumIsReady re-materialises the bundle on every call for struct bundles — result == null on a notnull type parameter boxes and is always false. Reintroduces the exact defect V2 was written to fixAssetBundleLoader.cs:45
M3MediumAssetHandle.value/isResolved are published without a fence the reader participates in. On ARM64 a reader can see isResolved == true with a stale null valueAssetHandle.cs:7-19
M4MediumReloadOne starts the rebuild on the calling thread — unlike Load, it is not wrapped in Task.Run, so a synchronous D3DCompile can run on the main threadHotReloadManager.cs:175
M5MediumDisposal is uncoordinated with in-flight work: HotReloadManager.Dispose clears its dictionaries without the lock; AssetManager.Dispose neither drains the channel nor awaits outstanding requestsAssetManager.cs:159, HotReloadManager.cs:211
M6MediumUnload bypasses RequestLock: unloading an in-flight asset throws, then the load completes anyway and takes a lease nobody returns. This is V2's open cancellation question, still unansweredAssetManager.cs:126
M7MediumRegistering a transcoder after a load of that type throws on the worker — which via B4 kills the manager permanently. “Thread-safe” here means “won't corrupt the dictionary”, not “order-independent”AssetManager.cs:39,222
M8MediumHot reload discards hot.BuildMetaData and never re-tracks, so an .hlsl that gains a new #include never registers that file until process restartHotReloadable.cs:33-46
L1LowSuccessful hot reload is logged at LogLevel.Error (copy-paste from the adjacent failure message)HotReloadManager.cs:226
L2LowA failed encode leaves a truncated .cka. Self-heals via the blanket catch, but write-to-temp-then-move would make “a .cka on disk is complete” an invariantAssetEncoder.cs:24-35
L3LowDebug.Assert is the only double-resolve guard, so in Release a duplicate publish silently overwrites a resolved handleAssetHandle.cs:16
L4Low*.cka artifacts are written beside the source files and are not in .gitignore, so a first run fills the art folder and dirties git statusAssetUtilities.cs:7, .gitignore
L5LowNo ConfigureAwait(false) anywhere, though CapriKit.IO uses it on all of its awaits. Harmless only while no SynchronizationContext exists — and not harmless for M412 awaits, 4 files
L6Lowpublic sealed record ExampleAssets ships in the NuGet package; AssetHandle's fields are internal protected with a public constructor, so external code can forge a resolved handleAssetBundleLoader.cs:66, AssetHandle.cs:7
-
- -
- Checked and found clean — recorded so these are not re-reviewed later -
    -
  • Envelope round trip. Field order in AssetEncoder matches both decode paths exactly; - SkipPayload advances by precisely payloadLength; endianness is symmetric; - SliceUnread correctly advances the outer reader past a sub-reader that under-reads; - ArrayPool.Rent over-allocates but SequenceReaders.Create(buffer, 0, length) bounds it.
    • -
  • Lock ordering. The only nestings are RequestLock → Cache.Lock and - RequestLock → TrackingLock, consistently. No deadlock exists today — but the invariant is written - down nowhere, and is one careless Unload inside a cache callback away from breaking.
    • -
  • AssetCache.Collect's TryEnter fast path. Correct. The - try/finally releases the lock on the early return, PendingDispose is monotone so nothing is - lost, and deferring a disposal by one frame is the intent. Worth a comment so nobody “fixes” it into a - blocking lock.
    • -
  • Disposing outside the lock. DisposeDrainedItems running outside Lock is - deliberate and correct — it is what makes an asset whose Dispose calls back into Unload - safe.
    • -
  • Path scoping. Watcher events are re-relativised to BasePath, matching the relative paths - VirtualFileSystemSpy records, so Dependents lookups hit. The .cka is written - through the raw file system, not the spy, so an asset never lists its own build output as a dependency — no - infinite rebuild loop.
    • -
  • Task.Run / FireAndForget shape. Binds the unwrapping overload, so - FireAndForget awaits the real work rather than a proxy. Correct.
    • -
-
-
-
- -
-
-

API: the boilerplate problem

-

Loading one asset costs four mentions of the same thing — the AssetId, the - Load line, the resolver.Get argument, and the bundle record member. Two of the four are removable - today. The third and fourth are inherent to “declare N loads, then build one object”, and only a source generator - collapses them.

- -
-
-

Today

- 
var id      = new AssetId(string.Empty, AssetFile);
-var builder = new AssetBundleBuilder(assetManager);
-var handle  = builder.Load<string, NoSettings>(id, default);
-var loader  = builder.Build(r => new TestBundle(r.Get(handle)));
-
-TestBundle? bundle = null;
-// poll assetManager.Update() until loader.IsReady(out bundle)
-
-
-

Proposed (P1–P5, no codegen)

- 
var bundle = assetManager.CreateBundle();
-var text   = bundle.Load<string>(AssetFile);
-var loader = bundle.Build(text, t => new TestBundle(t));
-
-using var assets = loader.Wait();   // bootstrap: pumps Update() itself
-
-
- -
- The honest limit -

For a realistic renderer — 3 shaders + 4 textures — these changes save about 22 characters per line - but the structure does not shrink. Worse, the factory is positional across same-typed assets, so the three - IVertexShader handles are mutually assignable: 144 wrong orderings compile and render silently - wrong. That is the real argument for the source generator (P8) — not brevity.

-
- -
- - - - - - - - - - - - -
IDChangeWinCostBreaking
P1Load<TAsset>(id) — erase TSettings at registrationBiggest per-character win; NoSettings leaves user code entirely~20 lnyes
P2AssetId: path first, key optional, implicit from FilePath onlyKills string.Empty and the silent argument-swap hazard~15 lnyes, 3 sites
P3assetManager.CreateBundle() instead of new AssetBundleBuilder(mgr)Discoverable from the object you already hold; V2 specified it3 lntrivially
P4AssetManager(ILoggerFactory, DirectoryPath) overloadCallers stop writing new FileSystem().ScopedTo(dir)4 lnno
P5Build overloads for arity 1–3 taking handles directlyResolver disappears; redemption becomes structurally impossible to get wrong~12 lnadditive
P6Wait() (block/pump) + IDisposable on the bundleCloses the two holes the code itself flags; makes the bundle the lifetime unit~40 lnyes
P7Per-bundle faulted state instead of the sticky channel errorFixes B4; a bad file stops poisoning every other load~15 lnno
P8[AssetBundle] source generatorOnly thing that collapses 4 mentions to 1 and kills the ordering hazard~300 lnadditive
-
- -

Stop the handle-passing overloads at arity 3. At arity 7, - Build(a,b,c,d,e,f,g, (a,b,c,d,e,f,g) => new X(...)) mentions each asset three times inside - Build versus once for r.Get(a). Keep both paths; document the crossover. - AssetHandleResolver does earn its keep — it is the capability token that makes redeeming a handle outside - materialization unrepresentable.

- -
- Considered and rejected — brevity that costs type safety -
    -
  • AssetHandle<T>.Value / IsLoaded. Removes one mention per asset. - Reject — this is the exact invariant both design docs exist to protect. A readable - .Value re-admits “not loaded yet” into every consumer's types and puts the per-frame check back - into every system.
    • -
  • Nullable bundle members with placeholder fallback. Reject for shaders — a - shader's interface is its identity, so no valid placeholder exists.
    • -
  • Ordered redemption (r.Next<T>()). Reject — silently couples - the factory to Load call order, strictly worse than explicit handle identity.
    • -
  • Reflection matching handles to record parameters. Reject — moves every arity and - type error to runtime, defeating the “compiler forces the constructor to match” payoff that the design docs - call the whole point of hardcoding requirements.
    • -
-
-
-
- -
-
-

Implementation size

-

1,195 lines across 13 files. A recommended set of seven changes projects to ~630 lines in 9 - files — a 47% reduction — with hot reload no longer in the shipping path.

- -
- - - - - - - - - - - - -
SubsystemFilesLinesShareVerdict
Hot reload228724%Largest, least tested, 5 defects — and dev-only by design
Orchestration (AssetManager)124921%Keep; trim with S5/S7
Envelope encode/decode323420%Reads every file twice; merge
Cache / refcounting118115%Deferred disposal may not be load-bearing
Bundles / handles211610%Keep; needs faulted state added back
Transcoder abstraction2928%The TSettings arity is the expensive half
Model + glue2363%—
Total131,195100%Projected after: ~630
-
- -
- - - - - - - - - - - - -
IDSimplificationLinesRiskCapability lost
S1Delete dead weight: ExampleAssets, ServiceCollectionExtensions, Unload, the discarded pruned list, the leak-throw in Cache.Dispose−51nonenone
S2One AssetEnvelope over byte[], not streams. Move dependencies before the payload so staleness reads only the prefix; merge the two decode methods that read the same file twice−141lownone — staleness gets faster
S3Shrink AssetCache: drop PendingDispose, Collect, and the TryEnter path; dispose on zero, outside the lock−106low–medworker-thread deferred dispose
S4Handle/bundle trim — delete the empty AssetBundleLoader base, collapse two fields to one object?; add ~8 lines back for the faulted state−15/+8nonenone
S5Drop the transcoder registry; pass the transcoder to Load. Keying on typeof(TAsset) silently forbids two transcoders for one type — negative convenience−30lowload-by-type-alone
S6Hot reload via closures instead of a class hierarchy, and opt-in behind a flag so shipping builds never construct it−155med–highreload throttling, weak pruning
S7Delete the TSettings type parameter; fold settings into the transcoder's Version−83mediumper-request settings
Tiered: S1+S2+S4 → 988 · +S3+S5 → 852 · +S6+S7 → ~630−573
-
- -

On S7, note the per-request settings capability is already largely illusory: Transcoders is keyed by - asset type so one type has exactly one transcoder, and ToEncodedFilePath does not include settings — so - loading one id with two different settings already ping-pongs rebuilds against the same .cka.

- -

Keep as is — load-bearing despite looking complex

-
    -
  • LightweightChannel + main-thread Update() drain. The threading model itself, and - the only place a transcoder's ID3D11DeviceContext step can legally run.
    • -
  • Outstanding: Dictionary<AssetId, List<AssetHandle>>. This is V2's fix for the - 1:N in-flight relationship. Collapsing it to one handle per id reintroduces the hang V2 was written to fix.
    • -
  • VirtualFileSystemSpy dependency capture. Four lines in the encoder buy the entire - #include staleness and hot-reload trigger story. This is the reuse model the rest of the pipeline should - imitate.
    • -
  • PutOrLease returning the winner and disposing the loser. Looks like paranoia; is not. - Load only dedupes inside the lock, and hot reload decodes concurrently with normal loads.
    • -
  • AssetHandle<T> as an empty generic subclass. A zero-cost compile-time type tag — - what makes “not yet loaded” unrepresentable. The runtime-Type alternative is longer and weaker.
    • -
  • HotSwap on the transcoder. Why no AssetRef<T> indirection layer is needed - at all — game code holds the object directly.
    • -
-
-
- -
-
-

Suggested order

-

A genuine sequence — each step makes the next one safe or cheaper.

-
    -
  1. B1 — make the solution build. -

    Nothing else can be verified end-to-end while the only real consumer is uncompilable. Decide here whether - AssetManagerExtensions is deleted or reshaped around the builder (P5 answers this).

    2. -
  2. B2, B3, B4 — the three criticals. -

    Two return-shaped fixes and one lease-counting fix. All three have executed repros, so each can be turned - into a regression test immediately.

    3. -
  3. Write the tests those repros became. -

    Second-run/up-to-date load, two handles for one id, and a failing load followed by ten Update()s. These are - the three paths the current test cannot reach, and they are what turn the rest of this list from review comments into a - safety net.

    4. -
  4. P1–P5 — the API shape, while call sites are still nearly zero. -

    At 0.1.0-alpha these breaking changes are free today and expensive later. P2 in particular closes a - silent-swap hazard that no amount of documentation fixes.

    5. -
  5. S1, S2, S4 — the zero-capability-loss deletions. -

    Takes the pipeline to ~988 lines and removes the double file read on every load. S2 is low-risk precisely because - B2 proves the staleness path has never run.

    6. -
  6. Decide hot reload's fate (H1–H5, S6). -

    It has never worked, so there is no regression to fear and full freedom to redesign. Make it opt-in first — five - lines — so it leaves the shipping mental model, then fix or rewrite behind that flag.

    7. -
  7. Settle the threading contract (H4, M3, M5). -

    The table below is the current state, not a specification. Pick one direction, write it in the XML docs, and delete the - // TODO comments that promise the other.

    8. -
-
-
- -
-
-

The threading model, as implemented

-

Not as documented — as it actually behaves today. Worth keeping as the module's reference page, since the - XML docs and the code currently disagree in three places.

-
- - - - - - - - - - - - - - - -
ClassFieldWritersReadersGuardVerdict
AssetManagerTranscodersanypoolconcurrent typesafe
Outstandingany + mainsameRequestLocksafe, but indexed by a worker-supplied key
Incomingpoolmainconcurrent + volatileerror slot sticky (B4)
AssetCacheEntries, RefCountanyanyLock alwayssafe
PendingDisposeanymainLock / TryEntercan contain a live object (B3)
AssetHandlevalue, isResolvedany, under lockmain, unlockednone on readunsynchronised publication (M3)
Ownerbuilder threadmainnonesame class of issue
HotReloadManagerTrackedmain in practicemainlookup onlyinner list iterated unlocked (H4)
Dependentsunder lockno lockinconsistentdoc contradicts code (H4)
isReloadingmain sets, pool clearsmainnonerace + latches true (H2)
PendingReloadspoolmainconcurrent typesafe
-
-

In one line: Load, Unload and RegisterTranscoder are genuinely callable from any - thread; RequestAsset, Encode and Decode run on the pool; and — despite the - documentation — MaterializeAsset, PutOrLease, Track, IsReady and - Get all run on the main thread. HotReloadable.Reload's synchronous prefix runs on the main thread - too, which is not intended (M4).

-
-
- -
-
-

Method. Four independent reviews ran in parallel against commit c9c853b, each reading the full - pipeline source plus its CapriKit.IO and CapriKit.Concurrency dependencies. Findings were merged by - defect and cross-checked against a manual read; the build failure and the two behaviour-changing claims about it were verified - directly. Six of the eight critical and high findings were confirmed by executing a repro; the rest are marked as reasoned from - the code. All temporary test files were removed and the working tree left clean.

-

Prior art. Supersedes AssetPipelineReview.md and - AssetPipelineRewriteReview.md, which reviewed earlier iterations. The design intent reviewed against is - AssetPipelineLoadingGroupsV2.md, whose model this code implements and whose open questions — cancellation, - hot-reload re-entry into a live bundle — remain open.

-
-
- - - diff --git a/Research/AssetPipelineVNextReviewContinued.html b/Research/AssetPipelineVNextReviewContinued.html deleted file mode 100644 index 017570b..0000000 --- a/Research/AssetPipelineVNextReviewContinued.html +++ /dev/null @@ -1,696 +0,0 @@ - - - - - -Asset Pipeline vNext — Follow-up - - - - -
-
-
CapriKit · feature/asset_pipeline · c7517e7
-

Asset Pipeline vNext — Follow-up

-

Verification of the fixes made in response to - AssetPipelineVNextReview.html, plus answers to the four design questions in - ResponseToFindingsOfAssetPipelineVNextReview.md. Items that are fixed get one line; - everything else is reported in full.

-
- Verified against 11 findings closed, 3 still open - Method read + 7 executed repros - Date 2026-08-15 - Working tree left clean, no source modified -
- -
-
Solution build
PASS
0 warnings, 0 errors
-
Hot reload
WORKS
first time, end‑to‑end
-
Closed
11
B1 B2 B4 H1 H2 H4 M2 M3 M4 M8 + L1 L4
-
Still open
3
B3 critical, M1+M6 high
-
New
4
N1–N4, all small
-
Answered
4
H3+H5, M1, M8, L5
-
-
-
- -
-
-

The headline

-

The two things the last report said had never run — the staleness fast path and - hot reload — both now run. That is the big result, and it changes what the rest of the plan should look like.

- -
- Executed, not inferred - 
[B2] first run  ready=True  encodes=1
-[B2] second run ready=True  SECOND-RUN ENCODES = 0   <-- loaded from the .cka
-
-[H]  initial load ready=True  text='Hello World'
-[H]  hot reload #1 swapped=True after 517ms  text='Goodbye World'
-[H]  hot reload #2 swapped=True after 532ms  text='Third Text'
-

Two consecutive reloads is the important half of the second result: it proves the isReloading - latch really clears, which is what H2 was about. 517 ms is the 500 ms debounce plus one frame.

-
- -

Against that, one critical survives. B3 is not fixed — the asset handed to a bundle is still - disposed on the very frame it resolves, and the guard added to prevent it contains the same type-confusion typo as H5. - The new per-bundle Unload is a good idea whose lease accounting does not balance, which is the M1 answer.

- -
- One consequence worth flagging -

The previous report argued that restructuring the envelope (S2) was low-risk because B2 proved the staleness - path had never executed. That argument is now void — the path is live and load-bearing. If you still want S2, - it now needs the second-run test to exist first.

-
-
-
- -
-
-

Closed — one line each

- -
- - - - - - - - - - - - - - - - - - - -
IDVerifiedResult
B1✓ fixedAssetManagerExtensions.cs was deleted rather than reshaped; dotnet build CapriKit.slnx now succeeds with 0 warnings and 0 errors across all 16 projects.
B2✓ fixedThe else at AssetManager.cs:121 closes the fall-through; a second manager over a directory that already has an up-to-date .cka performs 0 encodes and no longer throws KeyNotFoundException.
B4✓ fixedLightweightChannel's sticky error slot became a ConcurrentQueue<ExceptionDispatchInfo> drained one per read: a missing file makes Update() throw exactly 1 of 40 frames with the real FileNotFoundException, and healthy assets still load afterwards. Matches your intent of "fail loudly once". See N4 for a small hazard the new overload pair introduces.
H1✓ fixedStreamPipeWriterOptions(leaveOpen: true) plus the conditional dispose in finally; the encode→seek→decode round trip in HotReloadable.Reload completes. The mirror-image bug in AssetDecoder is still there — N3.
H2✓ fixedisReloading is volatile, and the assignment moved inside if (target != null) so the null-target case cannot latch. Two consecutive edits both hot-swapped, ~520 ms each.
H4✓ fixedThe candidate iteration in ReloadOne and the whole of DrainFileChanges now sit inside TrackingLock. One residual: PendingRebuilds is mutated inside the lock at :131 and outside it at :148–149. Harmless while both are main-thread — and the M8 answer below deletes the lock entirely, which resolves it.
M2✓ fixedA dedicated bool isReady replaces the result == null test, so struct bundles are materialised once.
M3✓ fixedvolatile bool isResolved written after value and read before Value gives a correct release/acquire pair. Reader side is IsReady → resolver.Get, which respects that order.
M4✓ fixedReloadOne wraps the reload in Task.Run, so the synchronous prefix of Reload (and any D3DCompile) is off the main thread.
M8✓ fixedHotSwapAction now carries hot.BuildMetaData.Dependencies and HotSwapPending calls RegisterFileDependencies, so a new #include is registered without a restart. Note it only ever adds — a removed #include keeps triggering rebuilds until restart. Cosmetic; mentioning it so it is not rediscovered.
L1✓ fixedSuccessful reload logs at Information.
L4✓ fixed*.cka is in .gitignore.
L6✓ partExampleAssets is gone from the shipped surface. AssetHandle's public constructor still lets external code forge a handle; low priority, listed for completeness.
M7waivedPer your note: transcoders are registered at start-up. Worth one sentence of XML doc on RegisterTranscoder saying so, since the failure mode is a worker-thread throw.
M5carriedStill open, as you asked. HotReloadManager.Dispose clears its four collections without TrackingLock and does not wait for an in-flight reload; AssetManager.Dispose drains neither Incoming nor the outstanding requests. Disposing while a load is in flight is undefined today.
-
-
-
- -
-
-

Still open

- -
-
B3Not fixed — the bundle still receives an asset that was disposed on the same frame
-
AssetManager.cs:171-181 (materializer still called per handle) · AssetCache.cs:40 (the new guard compares the wrong operands)
-
repro executedcorrectnessthreading
- -

Two things had to change and neither landed. The drain still materialises once per handle:

- 
var handles = Outstanding[id];
-foreach (var handle in handles)
-{
-    var asset = materializer();   // still once PER HANDLE
-    handle.Resolve(asset);
-}
- -

and the guard that was added to PutOrLease to make the second call harmless compares a - TAsset against an Entry wrapper — two unrelated types, so it is always - false, exactly like the H5 guard it was modelled on:

- 
if (object.ReferenceEquals(candidate, entry))   // TAsset  vs  Entry
-{
-    return candidate;
-}
-//                                  entry.Asset is what you meant
- - Failure scenario — executed - 
[B3-one-bundle]  ready=True same=True A.IsDisposed=True B.IsDisposed=True
-[B3-two-bundles] ready=True same=True IsDisposed=True
-

Both handles resolve to the same instance and that instance is already disposed when the bundle is handed over, - because PutOrLease queued the live object into PendingDispose and Cache.Collect() - — three lines later in the same Update() — disposed it. No exception is raised. For a - VertexShaderTranscoder that is a released ID3D11VertexShader handed to gameplay code.

- - Fix — four small edits, and they also settle M1 - 
// 1. AssetCache.PutOrLease: compare against the stored instance
-if (ReferenceEquals(candidate, entry.Asset)) { return candidate; }
-
-// 2. AssetCache: one place to add the leases for the other bundles
-public void AddLeases(AssetId id, int count)
-{
-    if (count <= 0) { return; }
-    lock (Lock)
-    {
-        ObjectDisposedException.ThrowIf(isDisposed, this);
-        if (Entries.TryGetValue(id, out var entry)) { entry.RefCount += count; }
-    }
-}
-
-// 3. AssetBundleBuilder: one handle per id per bundle
-private readonly Dictionary<AssetId, AssetHandle> Handles = [];
-
-public AssetHandle<TAsset> Load<TAsset, TSettings>(AssetId id, TSettings settings)
-    where TAsset : class
-{
-    if (Handles.TryGetValue(id, out var existing))
-    {
-        return existing as AssetHandle<TAsset>
-            ?? throw new InvalidOperationException($"{id} is already claimed in this bundle as another type");
-    }
-
-    var handle = assetManager.Load<TAsset, TSettings>(id, settings);
-    Handles.Add(id, handle);
-    return handle;
-}
-
-// 4. AssetManager.Update: materialise once, lease once per waiting bundle
-if (!Outstanding.Remove(id, out var handles)) { continue; }
-
-var asset = materializer();                 // exactly once
-Cache.AddLeases(id, handles.Count - 1);     // step 3 makes this the bundle count
-foreach (var handle in handles) { handle.Resolve(asset); }
-

Step 3 is what makes step 4 correct rather than approximately correct: once a builder can only produce one handle per - id, every handle in Outstanding[id] necessarily belongs to a different bundle, so - handles.Count is the number of bundles that will each call Return once.

-
- -
-
M1+M6The per-bundle Unload is the right unit, but the accounting does not balance and an in-flight unload corrupts the bundle
-
AssetManager.cs:140-158 · AssetBundle.cs:39-44 · AssetCache.cs:103
-
repro executedcorrectnessthreading
- -

This is the direct answer to "is this mechanism sound and thread-safe?" — the mutual exclusion is fine, - the arithmetic is not. Three separate problems.

- - 1 — leases are counted per handle, returns per distinct id -

AssetBundle.Assets is a HashSet<AssetId>, so Unload calls - Return once per distinct id. But leases are taken once per handle — - by Cache.TryLease on the cache-hit path and by the repeated PutOrLease on the load path. - The two only agree when every bundle holds exactly one handle per id:

- 
[B3-one-bundle] Unload ok
-  Dispose threw Exception: Cache will leak 1 entries that have not
-  been returned before the cache was disposed.
-

The invariant you want, and it is worth writing it into the XML doc of both Return and - PutOrLease, is: RefCount(id) equals the number of active bundles that contain - id. B3's steps 3 and 4 above establish exactly that on the load path; the cache-hit path in - Load gets it for free from step 3.

- - 2 — unloading an in-flight bundle throws, and then poisons the bundle - 
[M6] Unload while in flight threw InvalidOperationException:
-     Returned AssetId { Path = Hello.txt, Key =  } which was not found in the cache.
-[M6] after load completes: ready=True IsDisposed=False
-[M6] second Unload: ok            <-- silently did nothing
-[M6] Dispose threw Exception: Cache will leak 1 entries ...
-

One unhandled case produces four compounding failures: Return throws for an id that has not been - materialised yet; the loop aborts so the remaining ids are never returned either; the finally - sets IsActive = false anyway (N1), so the second Unload is a no-op and the leak is now - permanent; and the load completes afterwards and takes a lease nobody can ever return.

-

This is V2's open cancellation question arriving in code. There are two honest answers:

-
    -
  • Minimal — refuse it. Check Outstanding first and throw - before mutating anything, leaving IsActive true so a later Unload still works. - Three lines, no corruption, decision deferred. - 
    lock (RequestLock)
-{
-    if (!bundle.IsActive) { return; }
-    foreach (var id in bundle.Assets)
-    {
-        if (Outstanding.ContainsKey(id))
-        {
-            throw new InvalidOperationException(
-                $"Cannot unload a bundle while {id} is still loading. Unload after IsReady.");
-        }
-    }
-    // ... only now mutate
-}
    • -
  • Proper — let a dead owner cancel its own lease. Unload marks the bundle - inactive and skips ids that are still in Outstanding; the drain then counts only the handles whose - owner is still active, and if that count is zero it returns the single lease PutOrLease just took. - This needs handle.Owner to be known at Load time, which it is not today — - Owner is assigned in Build(). The structural fix is to let - CreateBundle() hand out the object that is the bundle identity, with - Build<TBundle> attaching the factory and returning a typed view over it. That also makes - "unload before Build" legal, which it currently is not.
    • -
- - 3 — two small mechanical issues in the method itself (N1, N2) -

See the next section.

- -

Thread-safety, separately: holding RequestLock across the whole of Unload - is correct — it makes Unload mutually exclusive with Load and with the - Update() drain, which is the guarantee you need. The lock ordering stays consistent - (RequestLock → Cache.Lock, never the reverse). Two things are unguarded but - benign today: AssetBundle.AssetSet is written by Build on one thread and read by - Unload on another with no barrier, and IsReady keeps handing out its cached - result after the bundle has been unloaded, so gameplay code can still reach disposed assets through a - loader it kept.

-
-
-
- -
-
-

New issues

-

All four are small. N1 and N2 are inside the new Unload; N3 and N4 came in with the H1 and - B4 fixes.

- -
- - - - - - - - -
IDSeverityFindingLocation
N1HighUnload's finally sets bundle.IsActive = false even when the loop threw partway through, so a bundle that failed to return its leases can never be unloaded again. Set it inside the body, after the returns succeed.AssetManager.cs:155
N2LowRequestLock.Enter() is inside the try, so if Enter ever throws, the finally calls Exit() without holding the lock and a SynchronizationLockException masks the real error. Either move Enter() above the try, or just use lock (RequestLock) { ... } — the whole method fits.AssetManager.cs:142-157
N3LowH1 was fixed in AssetEncoder but not in its mirror: using var input = inputStreamOverride ?? fileSystem.OpenRead(...) still disposes a stream the caller owns. Benign today only because HotReloadable passes a MemoryStream, whose Dispose is idempotent. Apply the same finally shape so the two sides stay symmetric.AssetDecoder.cs:23
N4LowThe new Write(T) / Write(ExceptionDispatchInfo) overload pair silently misroutes when T is ExceptionDispatchInfo: the non-generic overload wins, so a value written as a payload lands in the error queue and is rethrown at the reader. Verified: Ch<EDI>.Write(edi) -> queue=0, errors=1. Renaming one side to WriteError removes the whole class of confusion and reads better at the call site in AssetManager.Load.LightweightChannel.cs:27,32
-
- -
- Adjacent, outside this review's scope — JobResult<T>.Match is broken for value types -

Not part of the asset pipeline (only CapriKit.Tests.Tool uses it), but it sits next to the code you - changed for B4, so: for a value-type T, Result is default(T) rather than - null, so a failed result calls both callbacks:

- 
JobResult<int>.Failure("job", edi).Match(onSuccess, onFailure)
-  onSuccess called with 0   <-- should not happen
-  onFailure called
-

The Debug.Assert(Result == null ^ Exception == null) in the constructor fires for the same reason, so in - a Debug build this trips before it misbehaves. The usual fix is to store the discriminator explicitly - (private readonly bool isSuccess;) rather than inferring it from nullness, and make Match an - if/else so the two arms are exclusive by construction.

-
-
-
- -
-
-

Your questions

- - -
-

H3 / H5 — how should HotReloadManager and AssetCache work together? - Can the cache be the authority?

-
-

Yes, and it collapses both findings into one change. The bug underneath H3 and H5 is the same: - HotReloadManager is trying to answer a question it does not own. WeakReference.IsAlive - answers "has the GC collected this?", and the dedupe guard tries to answer "is this the instance we - already track?". The cache already answers both, exactly and without GC timing: it holds one entry per - AssetId, it knows the winning instance, and it knows the moment the last lease goes away.

- -

H3 is confirmed live, by the way — a hot swap runs happily against an unloaded, disposed asset:

- 
[H3] after Unload: IsDisposed=True text='Hello World'
-[H3] hot swapped a DISPOSED/unloaded asset = True (IsDisposed=True)
- - Step 1 — the cache reports eviction; it already knows - 
// AssetCache: true when the last lease went away
-public bool Return(AssetId id)
-{
-    lock (Lock)
-    {
-        ObjectDisposedException.ThrowIf(isDisposed, this);
-        if (!Entries.TryGetValue(id, out var entry)) { return false; }
-
-        entry.RefCount--;
-        if (entry.RefCount > 0) { return false; }
-
-        Entries.Remove(id);
-        PendingDispose.Enqueue(entry);
-        return true;
-    }
-}
-
-// AssetManager.Unload: eviction is now the one place tracking ends
-if (Cache.Return(id)) { HotReloadManager.UnTrack(id); }
-

You already wrote UnTrack — nothing calls it. This is its caller.

- - Step 2 — HotReloadable holds a strong reference, and Tracked stops being a list - 
private readonly TAsset Instance;   // was WeakReference<TAsset>
-// IsAlive, the candidate loop and both TODOs are deleted
-
-private readonly Dictionary<AssetId, HotReloadable> Tracked = [];
-
-public void Track<TAsset, TSettings>(Asset<TAsset, TSettings> asset, IAssetTranscoder<TAsset, TSettings> transcoder)
-    where TAsset : class
-    => Tracked[asset.Id] = new HotReloadable<TAsset, TSettings>(asset, transcoder);
-

H5 does not need fixing — it stops existing. The list was only there because the manager - could not tell which of several instances was the live one. The cache guarantees there is exactly one: - PutOrLease returns the winner and the loser is discarded. With one entry per id there is nothing to - dedupe, no candidate to choose, and no pruned list to forget to assign. The lifetime of the strong - reference is now bounded by UnTrack instead of by the GC, which is the whole point.

- - Step 3 — close the last-mile race -

One window remains: a rebuild is already on the pool when the bundle is unloaded. The swap must be validated on the - main thread, at the moment it is applied. Carry the target on the action so it can be compared:

- 
internal sealed record HotSwapAction(
-    AssetId Id, object Target, IReadOnlyList<Dependency> Dependencies,
-    Action PerformHotSwap, Action DiscardNewParts);
-
-// AssetCache
-public bool IsCurrent(AssetId id, object instance)
-{
-    lock (Lock) { return Entries.TryGetValue(id, out var e) && ReferenceEquals(e.Asset, instance); }
-}
-
-// HotSwapPending, main thread
-if (!Cache.IsCurrent(action.Id, action.Target))
-{
-    action.DiscardNewParts();   // nothing installed it; do not leak the rebuilt asset
-    continue;
-}
-

DiscardNewParts matters: HotSwap is what normally absorbs or frees - newParts, so a skipped swap leaks it otherwise.

- -

This depends on an ordering that is already correct in Update() — - Cache.Collect() at AssetManager.cs:184 runs before - HotReloadManager.Update() at :185. Evict-then-swap is what makes a stale swap detectable - rather than a race. Worth a one-line comment saying so, because reordering those two lines would silently reopen H3.

- -
- - - - - - - - - -
TodayWith the cache as authority
liveness meansWeakReference.IsAlive — GC timingthe cache still holds a lease — deterministic
Tracked valueList<HotReloadable> + candidate loop + 2 TODOsone HotReloadable
H5 dedupebroken ReferenceEquals across two typesnot needed
swap onto a dead assethappens (repro'd above)rejected by IsCurrent
size—roughly −35 lines
-
-
-
- - -
-

M8 — there is a lot of locking in HotReloadManager. Can it be simpler or more explicit? - Hot reload is rare, so should it use concurrent collections more?

-
-

Go the other way: delete the lock, do not add concurrent collections. Map the state first — - once you do, the reason the locking feels heavy is that it is guarding against a caller that does not exist.

- -
- - - - - - - - - - - -
StateWritten byRead byGuard actually needed
Trackedmain — Track ← MaterializeAsset ← Update()mainnone
Dependentsmain — Track and HotSwapPendingmainnone
PendingRebuildsmainmainnone
lastFileChangemainmainnone
FileChanceswatcher threadmainalready a concurrent queue
PendingReloadspoolmainConcurrentQueue ✓
isReloadingmain sets, pool clearsmainvolatile ✓
-
- -

Only the last three rows are genuinely cross-thread, and all three are already handled. TrackingLock - guards the first four — which never leave the main thread, because Track is reached only from - MaterializeAsset, which is reached only from the Update() drain. And that is not an - accident of the current code: materialisation has to happen on the main thread, because that is the only - place a transcoder's ID3D11DeviceContext work can legally run. It is a structural constraint, so it is - safe to lean on.

- -

So delete TrackingLock — four lock blocks — and replace it with something - that states the constraint instead of defending against its opposite:

- 
// AssetManager captures this once, in its constructor
-private readonly int MainThreadId = Environment.CurrentManagedThreadId;
-
-[Conditional("DEBUG")]
-private void AssertMainThread([CallerMemberName] string caller = "")
-    => Debug.Assert(Environment.CurrentManagedThreadId == MainThreadId,
-        $"{caller} must run on the thread that created the AssetManager.");
-

Call it at the top of Update, Collect, Track, UnTrack and - IsReady. It costs nothing in Release, it fires immediately and by name if the assumption ever breaks, - and it removes the contradiction the last report flagged — Track documented as - "thread-safe, multiple threads" while DrainFileChanges two methods later says "must be called single - threaded". This is also step 7 of the previous report's suggested order, done cheaply.

- -
- Why not concurrent collections -

Because they would be less safe while looking safer, and this class has already been bitten by exactly - that. ConcurrentDictionary<AssetId, List<HotReloadable>> makes the dictionary thread-safe - and leaves the inner List unguarded — that is H4's original shape, verbatim. And - Track needs lookup, mutate and RegisterFileDependencies to be atomic together; - concurrent collections give per-operation atomicity, never multi-step. Rarity is an argument for a plain lock over - a clever one, not for a lock-free type.

-
- -

Two side effects worth having: the H4 residual disappears (PendingRebuilds stops being inside the lock - in one place and outside it in another), and so does the current reliance on System.Threading.Lock - being reentrant — Track holds TrackingLock and calls - RegisterFileDependencies, which takes it again. That works, but it is a subtlety you no longer have to - know about. Do the H3 change first: with Tracked holding one entry instead of a list, there is barely - anything left for a lock to protect.

-
-
- - -
-

L5 — where am I missing ConfigureAwait(false), given that - FireAndForget already sets it?

-
-

All twelve awaits in the pipeline, in four files. FireAndForget's - ConfigureAwait(false) applies only to its own await of the outer task — it does not - propagate into the awaits inside RequestAsset or anything it calls. Each of these captures the ambient - SynchronizationContext independently.

- -
- - - - - - - - -
FileLinesWhat is awaited
AssetManager.cs114, 117, 128, 129TryDecodeBuildMetaData, Decode ×2, Encode
AssetEncoder.cs31, 34, 35, 64WritePayload, FlushAsync, CompleteAsync, encoder.Encode
AssetDecoder.cs29, 65input.ReadExactlyAsync ×2
HotReloadable.cs40, 43AssetEncoder.Encode, AssetDecoder.Decode
-
- -

None of them is a live bug today, and it is worth being clear about why, because it also tells you - when it stops being true. Every one of these paths is entered through Task.Run — - Load at AssetManager.cs:92 and, since the M4 fix, ReloadOne at - HotReloadManager.cs:185. On a pool thread SynchronizationContext.Current is - null, so every continuation already resumes on the pool. The M4 fix is what removed the one path that - could have hit this.

- -

It becomes load-bearing the moment any of these methods is awaited directly from a thread that has a - context — a WinForms or WPF host, or a custom game-loop synchronisation context. Since these ship as a NuGet - package you do not get to know the host. So: add it to all twelve, and rather than maintaining it by hand, turn on - CA2007 for the library projects so the compiler keeps it honest. CapriKit.IO already does - this by hand, which is exactly the sort of consistency that decays silently.

-
-
-
-
- -
-
-

Suggested order from here

-

Much shorter than last time. Steps 1 and 2 are the same edit set, which is the main reason to do them together.

-
    -
  1. B3 + M1 — the four edits. -

    Fix the PutOrLease operand, add AddLeases, dedupe in the builder, materialise once. This - is one coherent change that establishes a single invariant: RefCount(id) is the number of active - bundles holding id. Write that invariant in the XML docs while it is fresh.

    2. -
  2. M6 — pick minimal or proper, then N1 and N2. -

    If you take the minimal route, N1 becomes moot because nothing throws mid-loop any more. If you take the proper - route, do the CreateBundle() restructure first, since that is what makes Owner known at - Load time.

    3. -
  3. H3 + H5 — make the cache the authority. -

    Return returns bool, Unload calls UnTrack, strong reference, - Tracked becomes a single entry, IsCurrent guards the swap. Net negative lines, and it - deletes both TODOs in HotReloadManager.

    4. -
  4. M8 — delete TrackingLock, add AssertMainThread. -

    Cheapest after step 3, because there is much less state left to reason about.

    5. -
  5. Turn the repros into tests. -

    The seven listed at the bottom of this report are the paths AssetManagerTests still cannot reach. - The second-run test in particular is now protecting real behaviour rather than dead code.

    6. -
  6. Then N3, N4, L5, M5 — and only then revisit S1–S7. -

    Re-read S2's risk note before starting it: its justification was that the staleness path had never executed, - which is no longer true.

    7. -
-
-
- -
-
-

Method. Read of the full pipeline at c7517e7 plus the changed - CapriKit.Concurrency files, a full-solution build, and seven repros executed as temporary TUnit tests - against the real AssetManager: second-run staleness (B2), two handles in one bundle and in two bundles - (B3), unload-in-flight (M6), failed load followed by 40 frames (B4), hot reload twice in a row (H1/H2), and hot swap - after unload (H3). Terminal output quoted above is verbatim. The temporary test file and a scratch console project used - to confirm N4 were removed; the working tree is clean and no source file was modified.

-

Prior art. Continues AssetPipelineVNextReview.html and answers - ResponseToFindingsOfAssetPipelineVNextReview.md. Findings S0–S7 and P1–P8 from the previous - report are untouched and still stand, except that S2's risk assessment is now out of date.

-
-
- - - diff --git a/Research/HowIDidIt.md b/Research/HowIDidIt.md deleted file mode 100644 index 585289f..0000000 --- a/Research/HowIDidIt.md +++ /dev/null @@ -1,164 +0,0 @@ -# How I did it: HotReloadManagerV3 - -_Context: we were finishing the asset pipeline on the `feature/asset_pipeline` branch. Loading assets already -worked end-to-end, hot-reloading did not. Three earlier attempts (`HotReloadManager`, `HotReloadManagerV2`, -`HotReloadPipeline`) were abandoned. This document explains the choices behind the fourth attempt._ - -## The shape of the problem - -Hot-reloading is a small state machine that is awkward because its three steps have different threading rules: - -| Step | Where it must run | Why | -| --- | --- | --- | -| Notice a file changed | any thread (the watcher's) | the OS decides when | -| Rebuild + reload | any thread | it is slow, the main thread must not stall | -| Hot-swap | **main thread only** | `IAssetTranscoder.HotSwap` says so (it touches GPU resources) | - -So the object has to hop threads twice, and along the way we must never leave a lease behind in the -`AssetCache`, never dangle a reference, and never let a failure damage an asset that is already loaded. - -## The central decision: the main thread owns all the state - -The single idea that made this version simpler than the previous three is that **only the main thread owns -mutable state machine state**. `Update()` is the only place where work advances a step: - -``` -Update() - ├─ MarkStaleAssets() drain the file event queue → which assets are stale? - ├─ StartReloads() lease + Task.Run per stale asset → thread pool does the slow part - └─ FinishReloads() for every completed task: hot-swap, then return the lease -``` - -The background tasks are pure: they take an input, produce a `ReloadedAsset`, and touch nothing else. All the -bookkeeping (which asset is stale, which rebuild is running, which lease is held) lives in main-thread-only -fields. That is why there is only one lock in the class, and it guards only the two collections that `Track` -(any thread) genuinely shares with `Update`. - -The earlier attempts inverted this: the background task pushed results into a concurrent queue *and* returned -its own lease *and* was responsible for its own error handling. That spread the ownership of a lease across -two threads, which is exactly where V2's `// TODO: what happens to the lease?` came from. - -## Why I did not serialize to one asset at a time - -You offered to let me handle one asset at a time and suspected it would make things *more* complex. It would. -Running rebuilds in parallel is the naturally simple option here: - -- **Parallel** needs one dictionary, `InFlight: AssetId → Task`. Starting work is - `InFlight.Add(id, task)`, finishing it is "is the task completed?". -- **Serial** needs that *plus* a queue of assets waiting for their turn, plus a "am I currently busy?" flag, - plus a rule for what to do when the file of a queued asset changes again while it waits. - -Serial only removes concurrency I never had to reason about anyway, because the tasks share nothing. - -## The lease protocol - -The cache is the authority on whether an asset is still alive, so a rebuild has to hold a lease for its whole -duration. The invariant I settled on is: - -> **`TryStartReload` takes exactly one lease, and `FinishReload` returns exactly one lease, in a `finally`.** - -To make that hold, the lease is taken **synchronously on the main thread before the task is started**, not -inside the task: - -```csharp -// TrackedAsset -if (!cache.TryLease(Id, out var live)) { reload = null; return false; } -reload = Task.Run(() => Reload(live, fileSystem)); -``` - -This buys two things. The manager always knows whether a lease exists (no "did the task get one before it -threw?" question), and the task receives the live instance as a plain argument, so it never has to look at -shared state. A failed `TryLease` is not an error, it is how we learn that nobody uses the asset anymore, -which is also the moment we stop tracking it. - -Because `AssetCache.Return` takes only an `AssetId`, the manager can return the lease without knowing the -asset's type. That is what keeps the type erasure cheap. - -## Type erasure - -As you predicted in the shell, this needs an abstract non-generic base plus a generic subclass: - -- `TrackedAsset` — `AssetId` + `Dependencies` + `TryStartReload(...)`. This is what lives in the dictionaries. -- `TrackedAsset` — adds the `TSettings` and the `IAssetTranscoder`, and - is the only place that knows the real types. -- `ReloadedAsset` — the non-generic result: the new dependency list plus an `Action` that performs the swap. - The generic types are captured inside that closure, so the main thread can run the swap without knowing them. - -I deliberately did **not** store the whole `AssetBuildMetaData` on the tracked asset. Only `Settings` and -`Dependencies` are ever used again; the transcoder id and version are already fixed by the transcoder instance. - -## Debouncing - -An editor saving a file produces several change events (truncate, write, flush). Rebuilding on the first one -means reading a half-written file. So a rebuild only starts when nothing relevant changed for -`Debounce` (default 0.5s). - -Two details worth noting: - -- The timer is only reset by changes to files that some tracked asset actually depends on. Otherwise the asset - pipeline writing its own `.cka` build files would keep postponing rebuilds forever. -- The window is global rather than per-asset. Saving file A delays a pending rebuild of unrelated asset B by - half a second. For a development-only feature that is invisible, and a per-asset timestamp would mean another - dictionary. - -The constructor takes an optional `debounce` so tests can pass `TimeSpan.Zero` and stay deterministic instead -of sleeping. `Update` compares with `>=` precisely so that zero means "no debounce" and never depends on a -timer tick landing. - -## Failure handling - -The guarantee is that a failure never invalidates a live asset. That falls out of the design almost for free, -because nothing is mutated until the very last step: - -- **Rebuild/reload fails** → the task faults, we log it, the live asset was never touched. Its old contents - stay correct and the next file change tries again. -- **Hot-swap fails** → the transcoder made whatever it made of the instance; we can only log it. Throwing here - would take the game down over a development feature, which is the wrong trade. -- **Either way** → the `finally` returns the lease, so a broken shader can never leak cache entries. - -The rebuild writes into a `MemoryStream` instead of over the `.cka` file on disk, because another thread may -be reading that exact file to load the same asset. The cost is that the on-disk build stays stale after a hot -reload, so the next startup rebuilds the asset once. That seemed clearly better than corrupting a concurrent load. - -## Dispose - -`Dispose` stops the watcher, drops everything not yet started, and then **waits for the running rebuilds and -finishes them through the normal path** rather than abandoning them. That is not just tidiness: an abandoned -rebuild holds a lease (the cache would report leaked entries) and owns a freshly built asset whose native -resources are only cleaned up by `HotSwap`. Draining and completing them normally handles both, and reuses -`FinishReloads` verbatim. - -## Locking - -There is one lock, and the rules are: - -1. It guards `Tracked` and `Dependents` only — the two collections `Track` shares with `Update`. -2. **The hot-swap runs outside of it.** `HotSwap` is user code that may load another asset, which would take - the asset manager's `RequestLock` and then this lock. Holding this lock during the swap would be a real - deadlock, not a theoretical one. -3. Lock order in the system is `AssetManager.RequestLock → HotReloadManager.Lock → AssetCache.Lock`, and - nothing acquires them in the other direction, so there is no cycle. - -## Things I noticed while doing this - -Two of these will bite when you wire V3 into `AssetManager`: - -1. **`AssetManager.Dispose` disposes in the wrong order.** It calls `Cache.Dispose()` *before* - `HotReloadManager.Dispose()`. Since the hot reload manager can hold leases, the cache will throw - "will leak N entries", and the manager's `Cache.Return` will then throw `ObjectDisposedException`. The hot - reload manager must be disposed first. (`HotReloadManagerV3Tests` declares `cache` before `sut` so that C#'s - reverse disposal order gets this right, and the leak check then doubles as a lease-accounting assertion.) -2. **`AssetDecoder.Decode` checks that the `.cka` file exists even when you pass a stream override.** It works - today only because a loaded asset always has a build on disk. It is a trap for any future in-memory-only path. -3. **`HotReloadManagerV2.cs` did not compile** — it calls `RegisterFileDependencies`, which was never written. - That is committed in `HEAD`, so the branch did not build. I commented the call out with a `TODO` so I could - verify V3; deleting the abandoned experiments is your call. - -## What I left out - -- No cancellation of in-flight rebuilds. Saving a file twice quickly just rebuilds twice; the second result - wins because the swaps are ordered by the main thread. -- No coalescing of a rebuild with a concurrent first-time load of the same asset. The lease makes it safe, only - wasteful, and it is rare. -- Dead tracked assets are only cleaned up when a file change reveals that the cache dropped them. An asset that - is unloaded and never touched again leaves a small entry behind until then. diff --git a/Research/ResponseToFindingsOfAssetPipelineVNextReview.md b/Research/ResponseToFindingsOfAssetPipelineVNextReview.md deleted file mode 100644 index c051c9d..0000000 --- a/Research/ResponseToFindingsOfAssetPipelineVNextReview.md +++ /dev/null @@ -1,29 +0,0 @@ -I have worked through the findings you reported in C:\projects\csharp\CapriKit\research\AssetPipelineVNextReview.html. Read that first so that you are familiar with the issues and issue numbering. I have made substational changes to CapriKit.AssetPipelile and few changes to some tests and CapriKit.Concurrency to address most of the issues. - -Below are what I worked on and the questions I have for you. Write a new report html report (call it continued or something like that) and answer my questions. In your new report only give a short one line answer if the problem was fixed satisfactory. If the problem still exists or has created a new problem, report it as normal. - ---- - -I made changes to fix issues B1-B4, please verify. Note that for B4 I do not mind if one failed load kills the game, but I now added a way to handle that so that the game can at least throw a complete error message. - -I made changes to fix issues H1, H2 and H4, please verify - -For H3 "Hot reload can HotSwap an asset that Collect already disposed": I would like an example of how the HotReloadManager and Cache can work better together I think that will also help me with H5 The dedupe guard compares two different types, so it is always false. How to fix this can I use the cache as some sort of authority here? - -M1: I've changed how assets are disposed, this can now only be done for a bundle at a time. (see AssetManager.Unload). Of course in the tests nobody unloads the assets yet. But is this mechanism sound and thread-safe? - -M2, M3, M4: I think I fixed these, please check - -Keep M5 as a TODO in your next report. I need to look at that later. - -M6, I've changed how unloading works, is this now fixed? - -M7: Ignore that for now, users are supposed to initialize the asset manager with all transcoders on start-up - - -M8: I think I fxed this, but there is now a lot of locking going on in HotReloadManager, can we make this simpler or at least more explicit. Hot reloading is very rare so maybe its better to use the concurrent collection types more often? - - -L5: in which places am I missing `ConfigureAwait(false)` not that .FireAndForget sets `ConfigureAwait(false)` - -I have not looked at S0 to S7 diff --git a/source/CapriKit.AssetPipeline/Asset.cs b/source/CapriKit.AssetPipeline/Asset.cs index 83daf3c..4756f3f 100644 --- a/source/CapriKit.AssetPipeline/Asset.cs +++ b/source/CapriKit.AssetPipeline/Asset.cs @@ -9,9 +9,18 @@ namespace CapriKit.AssetPipeline; /// Optional key to a sub-resources in Path. public record AssetId(FilePath Path, string Key = ""); +/// +/// An asset, including its identifier and information on how it was built. +/// internal record Asset(AssetId Id, TAsset Value, AssetBuildMetaData BuildMetaData) where TAsset : class; +/// +/// Record of the exact transcoder, settings and files used to build the asset. +/// internal record class AssetBuildMetaData(Guid TranscoderId, int TranscoderVersion, TSettings Settings, IReadOnlyList Dependencies); +/// +/// A file used to build the asset and the date and time it was last changed +/// internal sealed record Dependency(FilePath File, DateTime Version); diff --git a/source/CapriKit.AssetPipeline/AssetBundle.cs b/source/CapriKit.AssetPipeline/AssetBundleLoader.cs similarity index 64% rename from source/CapriKit.AssetPipeline/AssetBundle.cs rename to source/CapriKit.AssetPipeline/AssetBundleLoader.cs index c34b06d..7e59e55 100644 --- a/source/CapriKit.AssetPipeline/AssetBundle.cs +++ b/source/CapriKit.AssetPipeline/AssetBundleLoader.cs @@ -2,6 +2,9 @@ namespace CapriKit.AssetPipeline; +/// +/// Use the builder to describe how an asset bundle should be built. +/// public sealed class AssetBundleBuilder { private readonly List Handles = []; @@ -12,6 +15,11 @@ internal AssetBundleBuilder(AssetManager assetManager) this.assetManager = assetManager; } + /// + /// Each asset that needs to loaded returns a handle that can then put used in the lambda + /// for to describe how + /// the asset bundle should actually be built. + /// public AssetHandle Load(AssetId id, TSettings settings) where TAsset : class { @@ -20,6 +28,14 @@ public AssetHandle Load(AssetId id, TSettings setting return handle; } + /// + public AssetHandle Load(AssetId id) + where TAsset : class + => Load(id, default); + + /// + /// Create a loader that is used to check the loading progress of the bundle. + /// public AssetBundleLoader Build(Func factory) where TBundle : notnull { @@ -34,7 +50,10 @@ public AssetBundleLoader Build(Func +/// Track the progress of loading the actual bundle and can be used to retrieve the actual bundle when finished. +/// +public abstract class AssetBundleLoader { private readonly HashSet AssetSet = []; @@ -44,8 +63,9 @@ public abstract class AssetBundle public IReadOnlySet Assets => AssetSet; } +/// public sealed class AssetBundleLoader(Func factory, IReadOnlyList handles) - : AssetBundle + : AssetBundleLoader where TBundle : notnull { private bool isReady; @@ -78,4 +98,6 @@ public bool IsReady([NotNullWhen(true)] out TBundle? value) } // TODO: Add a method to block and wait without eating all the CPU. + + // TODO: how can we put a sort of progress bar and progress information on this thing? } diff --git a/source/CapriKit.AssetPipeline/AssetHandle.cs b/source/CapriKit.AssetPipeline/AssetHandle.cs index c9d391b..92bd422 100644 --- a/source/CapriKit.AssetPipeline/AssetHandle.cs +++ b/source/CapriKit.AssetPipeline/AssetHandle.cs @@ -2,6 +2,9 @@ namespace CapriKit.AssetPipeline; +/// +/// Represents an asset that is in the progress of loading. +/// public abstract class AssetHandle(AssetId id) { public AssetId Id { get; } = id; @@ -10,7 +13,7 @@ public abstract class AssetHandle(AssetId id) private volatile bool isResolved; - internal AssetBundle? Owner { get; set; } + internal AssetBundleLoader? Owner { get; set; } internal bool IsResolved => isResolved; internal object? Value => value; @@ -22,9 +25,13 @@ internal void Resolve(object asset) } } +/// public sealed class AssetHandle(AssetId id) : AssetHandle(id) { } -public sealed class AssetHandleResolver(AssetBundle owner) +/// +/// Helper class for resolving loaded assets from their asset handle. +/// +public sealed class AssetHandleResolver(AssetBundleLoader owner) { public TValue Get(AssetHandle promise) { diff --git a/source/CapriKit.AssetPipeline/AssetManager.cs b/source/CapriKit.AssetPipeline/AssetManager.cs index e65ac4e..9406ea4 100644 --- a/source/CapriKit.AssetPipeline/AssetManager.cs +++ b/source/CapriKit.AssetPipeline/AssetManager.cs @@ -7,11 +7,14 @@ namespace CapriKit.AssetPipeline; +/// +/// Manages the building, loading, chaching, clean-up and hot-reloading of assets. +/// public sealed partial class AssetManager : IDisposable { private readonly ILogger Logger; private readonly ScopedFileSystem FileSystem; - private readonly AssetCache Cache; + private readonly AssetPool Cache; private readonly HotReloadManager HotReloadManager; private readonly ConcurrentDictionary Transcoders; @@ -24,7 +27,7 @@ public AssetManager(ILoggerFactory logger, ScopedFileSystem fileSystem) Logger = logger.CreateLogger(); FileSystem = fileSystem; Cache = new(); - HotReloadManager = new(logger, fileSystem); + HotReloadManager = new(logger, Cache, FileSystem); Transcoders = []; Incoming = new(); RequestLock = new(); @@ -34,7 +37,7 @@ public AssetManager(ILoggerFactory logger, ScopedFileSystem fileSystem) /// /// Register a transcoder for the given asset type. Registering a transcoders for a type /// that was already assigned a transcoder throws an exception. - /// Threading: thread-safe, multiple threads can register transcoders at the same time. + /// Threading: thread-safe, multiple threads can register transcoders at the same time. /// public void RegisterTranscoder(IAssetTranscoder transcoder) where TAsset : class @@ -48,7 +51,7 @@ public void RegisterTranscoder(IAssetTranscoder - /// Use to defines a bundle of assets to load + /// Use to defines a bundle of assets to load. /// Threading: thread-safe. /// public AssetBundleBuilder CreateBundle() @@ -58,7 +61,7 @@ public AssetBundleBuilder CreateBundle() /// /// Starts loading an asset. The asset will either be loaded from the cache, from disk, or rebuild and then loaded. - /// The caller gets a handle to be used in an which can be resolved + /// The caller gets a handle to be used in an which can be resolved /// to the actual asset when loading finishes using /// Threading: thread-safe, can be called from any thread concurrently. This method guarantees that the same asset /// is not loaded multiple times concurrently. @@ -137,7 +140,7 @@ private async Task RequestAsset(AssetId id, TSettings setting /// Threading: Unload updates the internal state of the bundle using a lock so that it is safe /// to unload the same bundle from multiple threads. /// - public void Unload(AssetBundle bundle) + public void Unload(AssetBundleLoader bundle) { try { @@ -181,14 +184,16 @@ public void Update() } } - Cache.Collect(); + Cache.DisposeReleased(); HotReloadManager.Update(); } public void Dispose() { - Cache.Dispose(); + // Dispose the hot-reload manager first so that it can release any reference to + // assets it might still hold. HotReloadManager.Dispose(); + Cache.Dispose(); } // Thread safe, only touches the file system and uses thread safe transcoder methods and properties. @@ -216,7 +221,7 @@ private static bool IsUpToDate(IAssetTranscoder -/// Cache of live assets. Methods are thread-safe and can be accessed concurrently. However, -/// cleaning-up unused asset using the must be done from the main thread. +/// Pool of keyed, reference-counted deduplicated live assets. +/// Most methods are thread-safe and can be accessed concurrently. However, +/// cleaning-up unused asset using the must be done from the main thread. /// -internal sealed partial class AssetCache : IDisposable +internal sealed partial class AssetPool : IDisposable { private sealed class Entry(AssetId id, object asset, int refCount) { @@ -79,7 +80,7 @@ public bool TryLease(AssetId id, [NotNullWhen(true)] out TAsset? asset) /// /// Returns a leased asset. If every user returned their asset it becomes collectable. Which happens in - /// . After calling return the caller must no longer reference the asset instance. + /// . After calling return the caller must no longer reference the asset instance. /// Threading: thread-safe. /// public void Return(AssetId id) @@ -109,7 +110,7 @@ public void Return(AssetId id) /// Disposes all assets that no longer have users. /// Threading: this method must only be called from the primary thread. /// - public void Collect() + public void DisposeReleased() { List? toDispose = null; diff --git a/source/CapriKit.AssetPipeline/HotReloadManager.cs b/source/CapriKit.AssetPipeline/HotReloadManager.cs index eff0862..d63b692 100644 --- a/source/CapriKit.AssetPipeline/HotReloadManager.cs +++ b/source/CapriKit.AssetPipeline/HotReloadManager.cs @@ -1,135 +1,156 @@ -using CapriKit.Concurrency.Async; using CapriKit.IO; using CapriKit.IO.Watchers; using Microsoft.Extensions.Logging; -using System.Collections.Concurrent; using System.Diagnostics; namespace CapriKit.AssetPipeline; /// -/// Facilitates hot reloading and hot swapping of assets. Tracks the files used to create an asset -/// and triggers a rebuild on file changes. Takes care of threading and only performs the final -/// hot swap when is called. +/// Rebuilds, reloads and hot-swaps tracked assets whenever one of the files they were built from changes. +/// Rebuilding and reloading happen on the thread pool, only the final hot-swap runs on the main thread +/// (see ). +/// A failed rebuild, reload or hot-swap only means that the asset keeps its current contents, it never +/// invalidates a live asset and never leaves a lease behind in the . +/// Threading: may be called from any thread, and +/// must be called from the main thread. /// internal sealed partial class HotReloadManager : IDisposable { - private static readonly TimeSpan MinWaitTime = TimeSpan.FromSeconds(0.5); private readonly ILogger Logger; - + private readonly AssetPool Cache; private readonly ScopedFileSystem FileSystem; private readonly IVirtualFileSystemWatcher Watcher; - private readonly FileSystemEventQueue FileChances; + private readonly FileSystemEventQueue FileChanges; + private readonly TimeSpan Debounce; - private readonly Lock TrackingLock; - private readonly Dictionary> Tracked; + // Guards the two collections that Track (any thread) and Update (main thread) share. + private readonly Lock Lock; + private readonly Dictionary Tracked; private readonly Dictionary> Dependents; - private readonly HashSet PendingRebuilds; - private readonly ConcurrentQueue PendingReloads; + // Only touched by the main thread. + private readonly HashSet Stale; + private readonly Dictionary> InFlight; + private long lastChange; - private long lastFileChange; - private volatile bool isReloading; + private bool isDisposed; - public HotReloadManager(ILoggerFactory logger, ScopedFileSystem fileSystem) + /// + /// The optional debounce is how long to wait after the last relevant file change before rebuilding. + /// Editors write their buffer in several steps, without a pause we would rebuild the same asset once per + /// step and read half-written files. If not provided, it will be set to 500 milliseconds. + /// + public HotReloadManager(ILoggerFactory loggerFactory, AssetPool cache, ScopedFileSystem fileSystem, TimeSpan? debounce = null) { - Logger = logger.CreateLogger(); - + Logger = loggerFactory.CreateLogger(); + Cache = cache; FileSystem = fileSystem; - Watcher = fileSystem.Watch(); - FileChances = new(Watcher); + Debounce = debounce ?? TimeSpan.FromSeconds(0.5); - TrackingLock = new(); + Lock = new(); Tracked = []; Dependents = []; + Stale = []; + InFlight = []; - PendingRebuilds = []; - PendingReloads = []; - - lastFileChange = Stopwatch.GetTimestamp(); - isReloading = false; + Watcher = FileSystem.Watch(); + FileChanges = new FileSystemEventQueue(Watcher); } /// - /// Registers an asset for tracking by the hot-reload system. - /// Threading: thread-safe, multiple threads can call this method and can even register - /// the same asset multiple times. This class figures out which instances are still relevant. + /// Registers an asset so that it is rebuilt, reloaded and hot-swapped whenever one of the files it was + /// built from changes. Tracking the same asset more than once is a no-op. + /// Threading: thread-safe, may be called from any thread at any time. /// public void Track(Asset asset, IAssetTranscoder transcoder) where TAsset : class { - // An asset can be registered multiple times - lock (TrackingLock) + lock (Lock) { - var reloadable = new HotReloadable(asset, transcoder); - if (Tracked.TryGetValue(asset.Id, out var assets)) - { - // TODO: this is broken!!!! - // Prevent adding the exact same instance multiple times - if (assets.Any(a => object.ReferenceEquals(a, asset))) { return; } - assets.Add(reloadable); - } - else - { - Tracked[asset.Id] = [reloadable]; - } + // After disposal we no longer listen for file changes, so tracking would only grow the maps. + if (isDisposed) { return; } - RegisterFileDependencies(asset.Id, asset.BuildMetaData.Dependencies); + // The asset manager materializes an asset once per outstanding handle, so the same asset arrives + // here several times. Everything we store comes from the build, so the first registration wins. + if (Tracked.ContainsKey(asset.Id)) { return; } + + var tracked = new TrackedAsset(asset, transcoder); + Tracked.Add(asset.Id, tracked); + RegisterDependencies(tracked); } } /// - /// Stops tracking an asset + /// Reacts to file changes, starts rebuilding the assets those files affect and hot-swaps the assets that + /// finished rebuilding. Every step is bounded work, the expensive rebuilding and reloading happens on the + /// thread pool so that the main thread only pays for the hot-swap itself. + /// Threading: must only be called from the main thread. /// - public void UnTrack(AssetId id) + public void Update() { - // Do not remove all references from dependents as that would require us to go through the entire collection - // but just forgetting it from tracked the asset will no longer be reloaded. - lock (TrackingLock) + if (isDisposed) { return; } + + MarkStaleAssets(); + + // Wait for the dust to settle so that a single save does not trigger a burst of rebuilds. + if (Stopwatch.GetElapsedTime(lastChange) >= Debounce) { - Tracked.Remove(id); + StartReloads(); } + + FinishReloads(); } /// - /// Checks which files required reloading, start the reloading process and hot swaps any asset that have reloaded - /// Threading: Unsafe, must only be called by the primary thread. Other threads can call other methods in this class. + /// Stops listening for file changes, abandons everything that has not started yet and finishes the + /// rebuilds that are already running so that their leases and freshly built data are handed back. + /// Threading: must only be called from the main thread. /// - public void Update() + public void Dispose() { - if (isReloading) + lock (Lock) { - return; + if (isDisposed) { return; } + isDisposed = true; } - DrainFileChanges(); - var elapsed = Stopwatch.GetElapsedTime(lastFileChange); - if (elapsed > MinWaitTime) + Watcher.Stop(); + Stale.Clear(); + + // The running rebuilds hold a lease and own freshly built data that only the main thread can dispose + // of, so instead of abandoning them we wait and then finish them through the regular path. + try { - ReloadOne(); + Task.WaitAll([.. InFlight.Values]); + } + catch (AggregateException) + { + // Failures are reported and cleaned up per asset by FinishReloads } - HotSwapPending(); + FinishReloads(); } /// - /// Drains the queue of file events and adds any assets that dependent on this file to PendingRebuilds - /// Threading: Unsafe, must be called single threaded because PendingRebuilds can only - /// be used by one thread at a time. + /// Marks every asset that depends on a changed file as stale. /// - private void DrainFileChanges() + private void MarkStaleAssets() { - lock (TrackingLock) + lock (Lock) { - while (FileChances.TryDequeue(out var @event)) + while (FileChanges.TryDequeue(out var change)) { - if (Dependents.TryGetValue(@event.File, out var dependents)) + if (!Dependents.TryGetValue(change.File, out var dependents)) { continue; } + + // Only changes we actually care about restart the debounce window, otherwise unrelated + // writes (such as the asset pipeline writing its own build files) could postpone a rebuild. + lastChange = Stopwatch.GetTimestamp(); + + foreach (var id in dependents) { - lastFileChange = Stopwatch.GetTimestamp(); - foreach (var id in dependents) + if (Stale.Add(id)) { - PendingRebuilds.Add(id); - LogPendingReload(Logger, @event.File, id); + LogAssetStale(Logger, change.File, id); } } } @@ -137,138 +158,152 @@ private void DrainFileChanges() } /// - /// Starts rebuilding the first asset in the set - /// Threading: Unsafe, must be called single threaded because the PendingRebuilds sets can only - /// be used by one thread at a time and the isReloading guard would also be confused. + /// Starts rebuilding and reloading every stale asset on the thread pool. /// - private void ReloadOne() + private void StartReloads() { - if (PendingRebuilds.Count == 0) { return; } - - var id = PendingRebuilds.First(); - PendingRebuilds.Remove(id); + if (Stale.Count == 0) { return; } - HotReloadable? target = null; - List? candidates; - - // Even though this method is single threaded, other methods that allow parallelism can - // touch Tracked so we need to put a lock around it. - lock (TrackingLock) + lock (Lock) { - if (!Tracked.TryGetValue(id, out candidates)) + foreach (var id in Stale.ToArray()) { - return; - } + // Let the running rebuild finish first. The asset stays stale so we pick it up again + // afterwards, which is exactly what we want because its files changed once more. + if (InFlight.ContainsKey(id)) { continue; } - // TODO: we allow users to add the same asset-id multiple times but we expect - // that (after a while) only one of the asset instances is still used by the engine. The - // others will be garbage collected eventually. Still, at this point in time we cannot be sure - // that the first (or last, or..) alive candidate is the one that will survive. - // How should we deal with that? - foreach (var candidate in candidates) - { - if (candidate.IsAlive) + Stale.Remove(id); + + if (!Tracked.TryGetValue(id, out var tracked)) { continue; } + + if (tracked.TryStartReload(Cache, FileSystem, out var reload)) + { + InFlight.Add(id, reload); + LogReloadStarted(Logger, id); + } + else { - target = candidate; - break; + // The cache is the authority on liveness: no entry means nobody uses this asset anymore. + UntrackAsset(tracked); + LogUntracked(Logger, id); } } } + } - // Not finding a target is normal, it means a file - // changed but the asset depending on it is no longer in use. - if (target != null) + /// + /// Hot-swaps every asset that finished rebuilding and returns the lease that its rebuild took. + /// + private void FinishReloads() + { + if (InFlight.Count == 0) { return; } + + foreach (var (id, reload) in InFlight.ToArray()) { - LogReloadStarted(Logger, id); - - isReloading = true; - Task.Run(() => target.Reload(FileSystem, PendingReloads) - .FireAndForget(ex => - { - LogReloadFailed(Logger, id, ex.SourceException); - isReloading = false; - }, - () => - { - LogReloadCompleted(Logger, id); - isReloading = false; - })); + if (!reload.IsCompleted) { continue; } + + InFlight.Remove(id); + FinishReload(id, reload); } } - /// - /// Hot swaps the assets that have been reloaded. - /// Threading: Unsafe, the contract from used here - /// requires that assets are only hot swapped on the main thread - /// - private void HotSwapPending() + private void FinishReload(AssetId id, Task reload) { - while (PendingReloads.TryDequeue(out var action)) + try { - try + if (!reload.IsCompletedSuccessfully) { - LogHotSwapStarted(Logger, action.Id); - action.PerformHotSwap(); + // Nothing was touched yet, so the asset simply keeps the contents it already had. + LogReloadFailed(Logger, id, reload.Exception!); + return; + } + + var reloaded = reload.Result; - RegisterFileDependencies(action.Id, action.Dependencies); + // Deliberately outside of the lock: the transcoder runs code we do not control here. + reloaded.HotSwap(); + + UpdateDependencies(id, reloaded.Dependencies); + LogHotSwapped(Logger, id); + } + catch (Exception ex) + { + // A transcoder that fails half-way leaves the asset in whatever state it made of it, all we can + // do is report it. The alternative, throwing, would take down the game over a development feature. + LogHotSwapFailed(Logger, id, ex); + } + finally + { + // Balances the lease that TryStartReload took, whether we managed to hot-swap or not. + Cache.Return(id); + } + } - LogHotSwapCompleted(Logger, action.Id); + /// + /// Replaces the dependencies of an asset with the ones its latest build read. + /// + private void UpdateDependencies(AssetId id, IReadOnlyList dependencies) + { + lock (Lock) + { + if (Tracked.TryGetValue(id, out var tracked)) + { + UnregisterDependencies(tracked); + tracked.Dependencies = dependencies; + RegisterDependencies(tracked); } - catch (Exception ex) + } + } + + // Threading: must be called while holding the lock + private void RegisterDependencies(TrackedAsset tracked) + { + foreach (var (file, _) in tracked.Dependencies) + { + if (!Dependents.TryGetValue(file, out var ids)) { - LogHotSwapFailed(Logger, action.Id, ex); + ids = []; + Dependents.Add(file, ids); } + + ids.Add(tracked.Id); } } - // Updates the dependents - private void RegisterFileDependencies(AssetId id, IReadOnlyList dependencies) + // Threading: must be called while holding the lock + private void UnregisterDependencies(TrackedAsset tracked) { - lock (TrackingLock) + foreach (var (file, _) in tracked.Dependencies) { - foreach (var dependency in dependencies) + if (Dependents.TryGetValue(file, out var ids) && ids.Remove(tracked.Id) && ids.Count == 0) { - var file = dependency.File; - if (Dependents.TryGetValue(file, out var ids)) - { - ids.Add(id); - } - else - { - ids = [id]; - Dependents.Add(file, ids); - } + Dependents.Remove(file); } } } - public void Dispose() + // Threading: must be called while holding the lock + private void UntrackAsset(TrackedAsset tracked) { - Watcher.Stop(); - Tracked.Clear(); - Dependents.Clear(); - PendingRebuilds.Clear(); - PendingReloads.Clear(); + UnregisterDependencies(tracked); + Tracked.Remove(tracked.Id); } - [LoggerMessage(Level = LogLevel.Information, Message = "Detected file change: {path}, affecting asset: {asset}")] - private static partial void LogPendingReload(ILogger logger, FilePath path, AssetId asset); + [LoggerMessage(Level = LogLevel.Information, Message = "Detected change in file: {file}, marking asset: {asset} as stale")] + private static partial void LogAssetStale(ILogger logger, FilePath file, AssetId asset); - [LoggerMessage(Level = LogLevel.Information, Message = "Reloading asset pending: {asset}")] + [LoggerMessage(Level = LogLevel.Information, Message = "Started rebuilding and reloading asset: {asset}")] private static partial void LogReloadStarted(ILogger logger, AssetId asset); - [LoggerMessage(Level = LogLevel.Information, Message = "Reloading asset completed: {asset}")] - private static partial void LogReloadCompleted(ILogger logger, AssetId asset); - - [LoggerMessage(Level = LogLevel.Error, Message = "Reloading asset failed: {asset}")] + [LoggerMessage(Level = LogLevel.Error, Message = "Rebuilding or reloading asset: {asset} failed, it keeps its current contents")] private static partial void LogReloadFailed(ILogger logger, AssetId asset, Exception exception); - [LoggerMessage(Level = LogLevel.Information, Message = "Hot-swapping asset started: {asset}")] - private static partial void LogHotSwapStarted(ILogger logger, AssetId asset); + [LoggerMessage(Level = LogLevel.Information, Message = "Hot-swapped asset: {asset}")] + private static partial void LogHotSwapped(ILogger logger, AssetId asset); - [LoggerMessage(Level = LogLevel.Information, Message = "Hot-swapping asset completed: {asset}")] - private static partial void LogHotSwapCompleted(ILogger logger, AssetId asset); - - [LoggerMessage(Level = LogLevel.Error, Message = "Hot-swapping asset failed: {asset}")] + [LoggerMessage(Level = LogLevel.Error, Message = "Hot-swapping asset: {asset} failed")] private static partial void LogHotSwapFailed(ILogger logger, AssetId asset, Exception exception); + + [LoggerMessage(Level = LogLevel.Information, Message = "Stopped tracking asset: {asset}, it is no longer in the cache")] + private static partial void LogUntracked(ILogger logger, AssetId asset); } diff --git a/source/CapriKit.AssetPipeline/HotReloadManagerV2.cs b/source/CapriKit.AssetPipeline/HotReloadManagerV2.cs deleted file mode 100644 index 6a55907..0000000 --- a/source/CapriKit.AssetPipeline/HotReloadManagerV2.cs +++ /dev/null @@ -1,230 +0,0 @@ -using CapriKit.Concurrency.Async; -using CapriKit.IO; -using CapriKit.IO.Watchers; -using Microsoft.Extensions.Logging; -using System.Collections.Concurrent; -using System.Diagnostics; - -namespace CapriKit.AssetPipeline; - -internal record ReloadResult(AssetId Id, IReadOnlyList NewDependencies, Action HotSwap); - -internal abstract record ReloadableV2 -{ - public abstract Task Reload(ConcurrentQueue resultQueue); -} - -internal record ReloadableV2(AssetId Id, AssetBuildMetaData Metadata, IAssetTranscoder Transcoder, IVirtualFileSystem FileSystem, AssetCache Cache) - : ReloadableV2 - where TAsset : class -{ - public override async Task Reload(ConcurrentQueue resultQueue) - { - if (Cache.TryLease(Id, out var cold)) - { - try - { - using var steam = new MemoryStream(); - // We store the encoded asset in memory instead of on disk to prevent - // touching the file while other threads are also working on it. - using var stream = new MemoryStream(); - await AssetEncoder.Encode(Id, Transcoder, Metadata.Settings, FileSystem, stream); - - stream.Seek(0, SeekOrigin.Begin); - var hot = await AssetDecoder.Decode(Id, Transcoder, FileSystem, stream); - - resultQueue.Enqueue(new ReloadResult(Id, hot.BuildMetaData.Dependencies, - () => - { - Transcoder.HotSwap(cold, hot.Value); - Cache.Return(Id); - })); - } - catch - { - Cache.Return(Id); - throw; - } - } - } -} - -internal sealed partial class HotReloadManagerV2 : IDisposable -{ - private static readonly TimeSpan MinWaitTime = TimeSpan.FromSeconds(0.5); - private readonly ILogger Logger; - private readonly AssetCache Cache; - private readonly ScopedFileSystem FileSystem; - private readonly IVirtualFileSystemWatcher Watcher; - private readonly FileSystemEventQueue FileChanges; - private readonly Dictionary Tracked; - private readonly Dictionary> Dependents; - - private readonly HashSet PendingRebuilds; - private readonly ConcurrentQueue PendingReloads; - - private readonly Lock Lock; - - private long lastFileChange; - - public HotReloadManagerV2(ILoggerFactory logger, AssetCache cache, ScopedFileSystem fileSystem) - { - Logger = logger.CreateLogger(); - Cache = cache; - FileSystem = fileSystem; - Lock = new(); - Tracked = []; - Dependents = []; - PendingRebuilds = []; - PendingReloads = []; - - Watcher = FileSystem.Watch(); - FileChanges = new(Watcher); - } - - - - public void Track(AssetId id, AssetBuildMetaData metadata, IAssetTranscoder transcoder) - where TAsset : class - { - lock (Lock) - { - // Track each asset only once - if (Tracked.TryGetValue(id, out var _)) { return; } - Tracked[id] = new ReloadableV2(id, metadata, transcoder, FileSystem, Cache); - - foreach (var dependency in metadata.Dependencies) - { - var file = dependency.File; - if (Dependents.TryGetValue(file, out var ids)) - { - ids.Add(id); - } - else - { - ids = [id]; - Dependents.Add(file, ids); - } - } - } - } - - public void Update() - { - DrainFileChanges(); - var elapsed = Stopwatch.GetElapsedTime(lastFileChange); - if (elapsed > MinWaitTime) - { - Rebuild(); - } - - HotSwapPending(); - } - - - /// - /// Drains the queue of file events and adds any assets that dependent on this file to PendingRebuilds - /// Threading: thread-safe - /// - private void DrainFileChanges() - { - lock (Lock) - { - while (FileChanges.TryDequeue(out var @event)) - { - if (Dependents.TryGetValue(@event.File, out var dependents)) - { - lastFileChange = Stopwatch.GetTimestamp(); - foreach (var id in dependents) - { - PendingRebuilds.Add(id); - LogPendingReload(Logger, @event.File, id); - } - } - } - } - } - - /// - /// Starts rebuilding the asset in the set - /// Threading: thread-safe - /// - private void Rebuild() - { - lock (Lock) - { - foreach (var id in PendingRebuilds) - { - if (Tracked.TryGetValue(id, out var reloadable)) - { - Task.Run(() => - { - LogReloadStarted(Logger, id); - reloadable.Reload(PendingReloads); // TODO: do I FireAndForget the outer, inner or both? - LogReloadCompleted(Logger, id); - }).FireAndForget(ex => - { - LogReloadFailed(Logger, id, ex.SourceException); - // TODO: consider error scenarios, especially - // what happens to the lease? - }); - } - } - PendingRebuilds.Clear(); - } - } - - /// - /// Hot swaps the assets that have been reloaded. - /// Threading: Unsafe, the contract from used here - /// requires that assets are only hot swapped on the main thread - /// - private void HotSwapPending() - { - while (PendingReloads.TryDequeue(out var reloadable)) - { - try - { - LogHotSwapStarted(Logger, reloadable.Id); - reloadable.HotSwap(); - - // TODO: this abandoned experiment does not compile, RegisterFileDependencies was never written. - // Commented out so that the project builds, see HotReloadManagerV3 for the version that works. - //RegisterFileDependencies(reloadable.Id, reloadable.NewDependencies); - - LogHotSwapCompleted(Logger, reloadable.Id); - } - catch (Exception ex) - { - LogHotSwapFailed(Logger, reloadable.Id, ex); - } - } - } - - public void Dispose() - { - Watcher.Stop(); - // TODO: drain in-progress reloads - } - - [LoggerMessage(Level = LogLevel.Information, Message = "Detected file change: {path}, affecting asset: {asset}")] - private static partial void LogPendingReload(ILogger logger, FilePath path, AssetId asset); - - [LoggerMessage(Level = LogLevel.Information, Message = "Reloading asset pending: {asset}")] - private static partial void LogReloadStarted(ILogger logger, AssetId asset); - - [LoggerMessage(Level = LogLevel.Information, Message = "Reloading asset completed: {asset}")] - private static partial void LogReloadCompleted(ILogger logger, AssetId asset); - - [LoggerMessage(Level = LogLevel.Error, Message = "Reloading asset failed: {asset}")] - private static partial void LogReloadFailed(ILogger logger, AssetId asset, Exception exception); - - [LoggerMessage(Level = LogLevel.Information, Message = "Hot-swapping asset started: {asset}")] - private static partial void LogHotSwapStarted(ILogger logger, AssetId asset); - - [LoggerMessage(Level = LogLevel.Information, Message = "Hot-swapping asset completed: {asset}")] - private static partial void LogHotSwapCompleted(ILogger logger, AssetId asset); - - [LoggerMessage(Level = LogLevel.Error, Message = "Hot-swapping asset failed: {asset}")] - private static partial void LogHotSwapFailed(ILogger logger, AssetId asset, Exception exception); -} diff --git a/source/CapriKit.AssetPipeline/HotReloadManagerV3.cs b/source/CapriKit.AssetPipeline/HotReloadManagerV3.cs deleted file mode 100644 index dea2312..0000000 --- a/source/CapriKit.AssetPipeline/HotReloadManagerV3.cs +++ /dev/null @@ -1,308 +0,0 @@ -using CapriKit.IO; -using CapriKit.IO.Watchers; -using Microsoft.Extensions.Logging; -using System.Diagnostics; - -namespace CapriKit.AssetPipeline; - -/// -/// Rebuilds, reloads and hot-swaps tracked assets whenever one of the files they were built from changes. -/// Rebuilding and reloading happen on the thread pool, only the final hot-swap runs on the main thread -/// (see ). -/// A failed rebuild, reload or hot-swap only means that the asset keeps its current contents, it never -/// invalidates a live asset and never leaves a lease behind in the . -/// Threading: may be called from any thread, and -/// must be called from the main thread. -/// -internal sealed partial class HotReloadManagerV3 : IDisposable -{ - private readonly ILogger Logger; - private readonly AssetCache Cache; - private readonly ScopedFileSystem FileSystem; - private readonly IVirtualFileSystemWatcher Watcher; - private readonly FileSystemEventQueue FileChanges; - private readonly TimeSpan Debounce; - - // Guards the two collections that Track (any thread) and Update (main thread) share. - private readonly Lock Lock; - private readonly Dictionary Tracked; - private readonly Dictionary> Dependents; - - // Only touched by the main thread. - private readonly HashSet Stale; - private readonly Dictionary> InFlight; - private long lastChange; - - private bool isDisposed; - - /// - /// The optional debounce is how long to wait after the last relevant file change before rebuilding. - /// Editors write their buffer in several steps, without a pause we would rebuild the same asset once per - /// step and read half-written files. - /// - public HotReloadManagerV3(ILoggerFactory loggerFactory, AssetCache cache, ScopedFileSystem fileSystem, TimeSpan? debounce = null) - { - Logger = loggerFactory.CreateLogger(); - Cache = cache; - FileSystem = fileSystem; - Debounce = debounce ?? TimeSpan.FromSeconds(0.5); - - Lock = new(); - Tracked = []; - Dependents = []; - Stale = []; - InFlight = []; - - Watcher = FileSystem.Watch(); - FileChanges = new FileSystemEventQueue(Watcher); - } - - /// - /// Registers an asset so that it is rebuilt, reloaded and hot-swapped whenever one of the files it was - /// built from changes. Tracking the same asset more than once is a no-op. - /// Threading: thread-safe, may be called from any thread at any time. - /// - public void Track(Asset asset, IAssetTranscoder transcoder) - where TAsset : class - { - lock (Lock) - { - // After disposal we no longer listen for file changes, so tracking would only grow the maps. - if (isDisposed) { return; } - - // The asset manager materializes an asset once per outstanding handle, so the same asset arrives - // here several times. Everything we store comes from the build, so the first registration wins. - if (Tracked.ContainsKey(asset.Id)) { return; } - - var tracked = new TrackedAsset(asset, transcoder); - Tracked.Add(asset.Id, tracked); - RegisterDependencies(tracked); - } - } - - /// - /// Reacts to file changes, starts rebuilding the assets those files affect and hot-swaps the assets that - /// finished rebuilding. Every step is bounded work, the expensive rebuilding and reloading happens on the - /// thread pool so that the main thread only pays for the hot-swap itself. - /// Threading: must only be called from the main thread. - /// - public void Update() - { - if (isDisposed) { return; } - - MarkStaleAssets(); - - // Wait for the dust to settle so that a single save does not trigger a burst of rebuilds. - if (Stopwatch.GetElapsedTime(lastChange) >= Debounce) - { - StartReloads(); - } - - FinishReloads(); - } - - /// - /// Stops listening for file changes, abandons everything that has not started yet and finishes the - /// rebuilds that are already running so that their leases and freshly built data are handed back. - /// Threading: must only be called from the main thread. - /// - public void Dispose() - { - lock (Lock) - { - if (isDisposed) { return; } - isDisposed = true; - } - - Watcher.Stop(); - Stale.Clear(); - - // The running rebuilds hold a lease and own freshly built data that only the main thread can dispose - // of, so instead of abandoning them we wait and then finish them through the regular path. - try - { - Task.WaitAll([.. InFlight.Values]); - } - catch (AggregateException) - { - // Failures are reported and cleaned up per asset by FinishReloads - } - - FinishReloads(); - } - - /// - /// Marks every asset that depends on a changed file as stale. - /// - private void MarkStaleAssets() - { - lock (Lock) - { - while (FileChanges.TryDequeue(out var change)) - { - if (!Dependents.TryGetValue(change.File, out var dependents)) { continue; } - - // Only changes we actually care about restart the debounce window, otherwise unrelated - // writes (such as the asset pipeline writing its own build files) could postpone a rebuild. - lastChange = Stopwatch.GetTimestamp(); - - foreach (var id in dependents) - { - if (Stale.Add(id)) - { - LogAssetStale(Logger, change.File, id); - } - } - } - } - } - - /// - /// Starts rebuilding and reloading every stale asset on the thread pool. - /// - private void StartReloads() - { - if (Stale.Count == 0) { return; } - - lock (Lock) - { - foreach (var id in Stale.ToArray()) - { - // Let the running rebuild finish first. The asset stays stale so we pick it up again - // afterwards, which is exactly what we want because its files changed once more. - if (InFlight.ContainsKey(id)) { continue; } - - Stale.Remove(id); - - if (!Tracked.TryGetValue(id, out var tracked)) { continue; } - - if (tracked.TryStartReload(Cache, FileSystem, out var reload)) - { - InFlight.Add(id, reload); - LogReloadStarted(Logger, id); - } - else - { - // The cache is the authority on liveness: no entry means nobody uses this asset anymore. - UntrackAsset(tracked); - LogUntracked(Logger, id); - } - } - } - } - - /// - /// Hot-swaps every asset that finished rebuilding and returns the lease that its rebuild took. - /// - private void FinishReloads() - { - if (InFlight.Count == 0) { return; } - - foreach (var (id, reload) in InFlight.ToArray()) - { - if (!reload.IsCompleted) { continue; } - - InFlight.Remove(id); - FinishReload(id, reload); - } - } - - private void FinishReload(AssetId id, Task reload) - { - try - { - if (!reload.IsCompletedSuccessfully) - { - // Nothing was touched yet, so the asset simply keeps the contents it already had. - LogReloadFailed(Logger, id, reload.Exception!); - return; - } - - var reloaded = reload.Result; - - // Deliberately outside of the lock: the transcoder runs code we do not control here. - reloaded.HotSwap(); - - UpdateDependencies(id, reloaded.Dependencies); - LogHotSwapped(Logger, id); - } - catch (Exception ex) - { - // A transcoder that fails half-way leaves the asset in whatever state it made of it, all we can - // do is report it. The alternative, throwing, would take down the game over a development feature. - LogHotSwapFailed(Logger, id, ex); - } - finally - { - // Balances the lease that TryStartReload took, whether we managed to hot-swap or not. - Cache.Return(id); - } - } - - /// - /// Replaces the dependencies of an asset with the ones its latest build read. - /// - private void UpdateDependencies(AssetId id, IReadOnlyList dependencies) - { - lock (Lock) - { - if (Tracked.TryGetValue(id, out var tracked)) - { - UnregisterDependencies(tracked); - tracked.Dependencies = dependencies; - RegisterDependencies(tracked); - } - } - } - - // The three methods below must be called while holding the lock. - - private void RegisterDependencies(TrackedAsset tracked) - { - foreach (var (file, _) in tracked.Dependencies) - { - if (!Dependents.TryGetValue(file, out var ids)) - { - ids = []; - Dependents.Add(file, ids); - } - - ids.Add(tracked.Id); - } - } - - private void UnregisterDependencies(TrackedAsset tracked) - { - foreach (var (file, _) in tracked.Dependencies) - { - if (Dependents.TryGetValue(file, out var ids) && ids.Remove(tracked.Id) && ids.Count == 0) - { - Dependents.Remove(file); - } - } - } - - private void UntrackAsset(TrackedAsset tracked) - { - UnregisterDependencies(tracked); - Tracked.Remove(tracked.Id); - } - - [LoggerMessage(Level = LogLevel.Information, Message = "Detected change in file: {file}, marking asset: {asset} as stale")] - private static partial void LogAssetStale(ILogger logger, FilePath file, AssetId asset); - - [LoggerMessage(Level = LogLevel.Information, Message = "Started rebuilding and reloading asset: {asset}")] - private static partial void LogReloadStarted(ILogger logger, AssetId asset); - - [LoggerMessage(Level = LogLevel.Error, Message = "Rebuilding or reloading asset: {asset} failed, it keeps its current contents")] - private static partial void LogReloadFailed(ILogger logger, AssetId asset, Exception exception); - - [LoggerMessage(Level = LogLevel.Information, Message = "Hot-swapped asset: {asset}")] - private static partial void LogHotSwapped(ILogger logger, AssetId asset); - - [LoggerMessage(Level = LogLevel.Error, Message = "Hot-swapping asset: {asset} failed")] - private static partial void LogHotSwapFailed(ILogger logger, AssetId asset, Exception exception); - - [LoggerMessage(Level = LogLevel.Information, Message = "Stopped tracking asset: {asset}, it is no longer in the cache")] - private static partial void LogUntracked(ILogger logger, AssetId asset); -} diff --git a/source/CapriKit.AssetPipeline/HotReloadPipeline.cs b/source/CapriKit.AssetPipeline/HotReloadPipeline.cs deleted file mode 100644 index 7b5122d..0000000 --- a/source/CapriKit.AssetPipeline/HotReloadPipeline.cs +++ /dev/null @@ -1,77 +0,0 @@ -//using CapriKit.Concurrency.Primitives; - -//namespace CapriKit.AssetPipeline; - -//internal abstract record HotSwapRecipe -//{ -// public abstract void HotSwap(); -//} - - -//internal abstract record ReloadRecipe -//{ -// public abstract Task Reload(); -//} - -//internal abstract record ReloadRecipe(AssetId Id, AssetBuildMetaData Metadata, IAssetTranscoder Transcoder) -// : ReloadRecipe -// where TAsset : class -//{ -// public override Task Reload() -// { - -// } -//} - - -//internal sealed class HotReloadPipeline -//{ -// private readonly LightweightChannel WaitingForRebuild = new(); -// private readonly LightweightChannel WaitingForHotSwap = new(); - -// private volatile bool isEnabled = true; -// private volatile bool isWorking = false; -// private Task? reloadTask = null; -// private Lock StateLock = new Lock(); - -// public bool TryEnter(ReloadRecipe recipe) -// { -// lock (StateLock) -// { -// if (!isEnabled) { return false; } - -// WaitingForRebuild.Write(recipe); -// return true; -// } -// } - -// public void Update() -// { -// lock (StateLock) -// { -// if (isWorking) -// { -// if (reloadTask != null && reloadTask.IsCompletedSuccessfully) -// { -// WaitingForHotSwap.Write(reloadTask.Result); -// } -// } -// else -// { - -// if (WaitingForRebuild.TryRead(out var rebuild)) -// { -// reloadTask = rebuild.Reload(); -// reloadTask.Start(); // TODO: is this necessary? -// isWorking = true; -// } -// } -// } - - -// while (WaitingForHotSwap.TryRead(out var hotswap)) -// { -// hotswap.HotSwap(); -// } -// } -//} diff --git a/source/CapriKit.AssetPipeline/HotReloadable.cs b/source/CapriKit.AssetPipeline/HotReloadable.cs deleted file mode 100644 index 9943f89..0000000 --- a/source/CapriKit.AssetPipeline/HotReloadable.cs +++ /dev/null @@ -1,47 +0,0 @@ -using CapriKit.IO; -using System.Collections.Concurrent; - -namespace CapriKit.AssetPipeline; - -internal sealed record HotSwapAction(AssetId Id, IReadOnlyList Dependencies, Action PerformHotSwap); - -internal abstract class HotReloadable(AssetId id) -{ - public AssetId Id { get; } = id; - public abstract bool IsAlive { get; } - - public abstract Task Reload(IVirtualFileSystem fileSystem, ConcurrentQueue hotSwapActionQueue); -} - -internal sealed class HotReloadable : HotReloadable - where TAsset : class -{ - private readonly WeakReference Instance; - private readonly TSettings Settings; - private readonly IAssetTranscoder Transcoder; - - public HotReloadable(Asset asset, IAssetTranscoder transcoder) - : base(asset.Id) - { - Instance = new WeakReference(asset.Value); - Settings = asset.BuildMetaData.Settings; - Transcoder = transcoder; - } - - public override bool IsAlive => Instance.TryGetTarget(out var _); - - public override async Task Reload(IVirtualFileSystem fileSystem, ConcurrentQueue hotSwapActionQueue) - { - if (!Instance.TryGetTarget(out var cold)) { return; } - - // We store the encoded asset in memory instead of on disk to prevent - // touching the file while other threads are also working on it. - using var stream = new MemoryStream(); - await AssetEncoder.Encode(Id, Transcoder, Settings, fileSystem, stream); - - stream.Seek(0, SeekOrigin.Begin); - var hot = await AssetDecoder.Decode(Id, Transcoder, fileSystem, stream); - - hotSwapActionQueue.Enqueue(new HotSwapAction(Id, hot.BuildMetaData.Dependencies, () => Transcoder.HotSwap(cold, hot.Value))); - } -} diff --git a/source/CapriKit.AssetPipeline/IAssetTranscoder.cs b/source/CapriKit.AssetPipeline/IAssetTranscoder.cs index 8e885ec..8c76b7d 100644 --- a/source/CapriKit.AssetPipeline/IAssetTranscoder.cs +++ b/source/CapriKit.AssetPipeline/IAssetTranscoder.cs @@ -4,8 +4,8 @@ namespace CapriKit.AssetPipeline; /// -/// Interface for classes that builds assets (such as texture, models and sound effects) and load them -/// when the program needs them. Implementers must ensure that all methods are thread-safe. +/// Interface for classes that build assets (such as texture, models and sound effects) and load them +/// when the program needs them. Implementers must take care that most access needs to be thread-safe. /// public interface IAssetTranscoder { @@ -20,32 +20,30 @@ public interface IAssetTranscoder : IAssetTranscoder // Asynchronous since we expect the encoder to read external files /// - /// Loads the raw asset data from the file system and build/encodes it into a format optimized for loading and handling in - /// an interactive simulation. Encoding happens asynchronously and can happen on any thread. + /// Loads the raw asset data from the file system and build/encodes it into a format optimized for loading. + /// Threading: thread-safe, encoding happens asynchronously and can happen on any thread. /// public Task Encode(AssetId id, TSettings settings, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer); - // Synchronous by design: the envelope owns all file IO and hands the decoder an - // in-memory payload. The reader's buffer is only valid for the duration of the call, - // decoders must copy out anything they want to keep. - /// - /// Decodes the file created the into an object by reading bytes from the given reader. Though decoding - /// itself is synchronous, it can happen as part of a multi-threaded or async operation. + /// Decodes the file created the into an object by reading bytes from the given reader. + /// Threading: thread-safe, Though decoding itself is synchronous, it can happen as part of + /// a multi-threaded or async operation. /// public TAsset Decode(AssetId id, TSettings settings, ref SequenceReader reader); - /// - /// Encodes the settings required to encode/decode the asset into the stream. Though this is by - /// itself synchronous, it can happen as part of a multi-threaded or async operation. + /// Encodes the settings required to encode/decode the asset into the stream. + /// Threading: thread-safe, though this is by itself a synchronous action, + /// it can happen as part of a multi-threaded or async operation. /// public void WriteSettings(TSettings settings, IBufferWriter writer); /// - /// Decodes the settings required to encode/decode the asset into the stream. Though this is by - /// itself synchronous, it can happen as part of a multi-threaded or async operation. + /// Decodes the settings required to encode/decode the asset into the stream. + /// Threading: Though this is by itself synchronous, + /// it can happen as part of a multi-threaded or async operation. /// public TSettings ReadSettings(ref SequenceReader reader); @@ -55,6 +53,7 @@ public interface IAssetTranscoder : IAssetTranscoder /// already hold references to. The transcoder is responsible for cleaning-up any /// orphaned resources. After calling this method must no longer /// be used or referenced. + /// Threading: must be called by the main thread. /// void HotSwap(TAsset instance, TAsset newParts); } diff --git a/source/CapriKit.AssetPipeline/README.md b/source/CapriKit.AssetPipeline/README.md new file mode 100644 index 0000000..349ca2e --- /dev/null +++ b/source/CapriKit.AssetPipeline/README.md @@ -0,0 +1,51 @@ +# CapriKit.AssetPipeline + +Pluggable asset pipeline that provides multi-threaded building and loading and hot-reloading of assets. + +## Usage + +1. Create the asset manager and register a transcoder, a class that knows how to build and load one asset type: +``` +var assetManager = new AssetManager(LoggerFactory, ScopedFileSystem); +assetManager.RegisterTranscoder(new VertexShaderTranscoder(GraphicsDevice)); +``` + +2. Define a bundle that reflects the strongly typed assets that you want to load. +``` +record UIBundle(PixelShader Shader, Texture2D Font); +``` + +3. Use the asset manager to create a bundle builder and start building (if needed) and loading assets. +``` +var builder = assetManager.CreateBundle(); +var shaderHandle = builder.Load(new AssetId("./shader.hlsl")); +var fontHandle = builder.Load(new AssetId("./"robo.png")); +``` +4. Use the obtained handles to describe how the bundle can be construct once building and loading finishes. +``` +var loader = builder.Build(r => new UIBundle(r.get(shaderHandle), r.get(fontHandle)); +``` + +5. Do not forget to update the asset manager each frame so that it can manage the loading tasks and other bookkeeping. +``` +assetManager.Update(); +``` + +6. Check each frame if loading finished and obtain your strongly typed bundle +``` +if (loader.isReady(out var bundle)) +{ + // Yay! +} +``` + +7. If you no longer need the assets, or if the program exists, return the bundle so the assets can be disposed. Loading and unloading uses reference counting to ensure that an asset is only truly disposed of it nobody references it anymore. To ensure everyone correctly unloads their assets an exception will be thrown if the `AssetManager` (and underlying `AssetPool`) are disposed without first unloading all asset bundles. +``` +assetManager.Unload(bundle); +``` + +## Hot reloading +Hot reloading happens automatically if the files that were used to build the asset are present and change. + +## Implementation +The AssetPool ensures that files diff --git a/source/CapriKit.AssetPipeline/TODO.md b/source/CapriKit.AssetPipeline/TODO.md deleted file mode 100644 index 420adc0..0000000 --- a/source/CapriKit.AssetPipeline/TODO.md +++ /dev/null @@ -1,7 +0,0 @@ -# TODO - -- What about using channels? -- What if reloading builds to a random file, so you don't need syncing ever, and the main thread triggers moving the file to the right place. -- Instead of documentation -> Throw early -- Look at bot-review -- How top opinionate more diff --git a/source/CapriKit.AssetPipeline/TrackedAsset.cs b/source/CapriKit.AssetPipeline/TrackedAsset.cs index 0c1d6da..134c725 100644 --- a/source/CapriKit.AssetPipeline/TrackedAsset.cs +++ b/source/CapriKit.AssetPipeline/TrackedAsset.cs @@ -13,7 +13,7 @@ namespace CapriKit.AssetPipeline; internal sealed record ReloadedAsset(IReadOnlyList Dependencies, Action HotSwap); /// -/// Everything the needs to rebuild a single asset. The asset and settings +/// Everything the needs to rebuild a single asset. The asset and settings /// types are erased so that assets of every type can live in one collection. /// internal abstract class TrackedAsset(AssetId id, IReadOnlyList dependencies) @@ -33,7 +33,7 @@ internal abstract class TrackedAsset(AssetId id, IReadOnlyList depen /// caller owns that lease and must return it once has completed. /// Threading: main thread only. /// - public abstract bool TryStartReload(AssetCache cache, IVirtualFileSystem fileSystem, [NotNullWhen(true)] out Task? reload); + public abstract bool TryStartReload(AssetPool cache, IVirtualFileSystem fileSystem, [NotNullWhen(true)] out Task? reload); } /// @@ -50,7 +50,7 @@ public TrackedAsset(Asset asset, IAssetTranscoder? reload) + public override bool TryStartReload(AssetPool cache, IVirtualFileSystem fileSystem, [NotNullWhen(true)] out Task? reload) { // The lease pins the live instance for the entire rebuild, so the background thread never has to // wonder whether the object it is going to hot-swap into still exists. diff --git a/source/CapriKit.Tests/AssetPipeline/AssetManagerTests.cs b/source/CapriKit.Tests/AssetPipeline/AssetManagerTests.cs index 42db527..e4c5937 100644 --- a/source/CapriKit.Tests/AssetPipeline/AssetManagerTests.cs +++ b/source/CapriKit.Tests/AssetPipeline/AssetManagerTests.cs @@ -36,15 +36,17 @@ public async Task LoadAsset() var logger = NullLoggerFactory.Instance; var fileSystem = new FileSystem().ScopedTo(WorkingDirectory); + await fileSystem.WriteAllText(AssetFile, "Hello World"); + var assetManager = new AssetManager(NullLoggerFactory.Instance, fileSystem); - var transcoder = new TestTranscoder(); + var transcoder = new TextTranscoder(); assetManager.RegisterTranscoder(transcoder); var id = new AssetId(AssetFile); var builder = assetManager.CreateBundle(); - var handle = builder.Load(id, default); + var handle = builder.Load(id, default); var loader = builder.Build(resolver => new TestBundle(resolver.Get(handle))); TestBundle? bundle = null; @@ -56,11 +58,11 @@ await Assert.That(() => .Eventually(v => v.IsTrue(), TimeSpan.FromSeconds(5)); await Assert.That(bundle).IsNotNull(); - await Assert.That(bundle.Text).IsEqualTo(TranscoderText); + await Assert.That(bundle.Asset.Text).IsEqualTo(TranscoderText); // Load again to verify loading the same thing twice gives us the cached value var altBuilder = new AssetBundleBuilder(assetManager); - var altHandle = altBuilder.Load(id, default); + var altHandle = altBuilder.Load(id, default); var altLoader = altBuilder.Build(resolver => new TestBundle(resolver.Get(altHandle))); TestBundle? altBundle = null; @@ -72,27 +74,34 @@ await Assert.That(() => .Eventually(v => v.IsTrue(), TimeSpan.FromSeconds(5)); await Assert.That(altBundle).IsNotNull(); - await Assert.That(altBundle.Text).IsSameReferenceAs(bundle.Text); + await Assert.That(altBundle.Asset).IsSameReferenceAs(bundle.Asset); + + // TODO: test the dispose path. Check the errors if assets are not returned and that returning both bundles correctly disposes everything. } +} - private record TestBundle(string Text); +internal record TestBundle(TextAsset Asset); - private class TestTranscoder() : NoSettingsTranscoder(Guid.Parse("{AC2D4E77-0D98-43B2-B1D2-35B0E9F5742B}"), 1) +internal sealed class TextAsset(string text) +{ + public string Text { get; set; } = text; +} + +internal class TextTranscoder() : NoSettingsTranscoder(Guid.Parse("{6E4A1D0C-1F73-4C4E-9D2E-0B7F5C6A9E31}"), 1) +{ + public override TextAsset Decode(AssetId id, ref SequenceReader reader) { - public override string Decode(AssetId id, ref SequenceReader reader) - { - return reader.ReadString(); - } + return new TextAsset(reader.ReadString()); + } - public override async Task Encode(AssetId id, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer) - { - var text = await fileSystem.ReadAllText(id.Path); - writer.Write(text); - } + public override async Task Encode(AssetId id, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer) + { + var text = await fileSystem.ReadAllText(id.Path); + writer.Write(text); + } - public override void HotSwap(string instance, string newParts) - { - throw new NotImplementedException(); - } + public override void HotSwap(TextAsset instance, TextAsset newParts) + { + instance.Text = newParts.Text; } } diff --git a/source/CapriKit.Tests/AssetPipeline/HotReloadManagerTests.cs b/source/CapriKit.Tests/AssetPipeline/HotReloadManagerTests.cs index 4200c26..9cf12be 100644 --- a/source/CapriKit.Tests/AssetPipeline/HotReloadManagerTests.cs +++ b/source/CapriKit.Tests/AssetPipeline/HotReloadManagerTests.cs @@ -1,15 +1,99 @@ using CapriKit.AssetPipeline; using CapriKit.IO; using Microsoft.Extensions.Logging.Abstractions; +using System.Buffers; namespace CapriKit.Tests.AssetPipeline; internal class HotReloadManagerTests { + private static readonly FilePath AssetFile = new("Hello.txt"); + private static readonly TimeSpan NoDebounce = TimeSpan.Zero; + [Test] - public async Task Foo() + public async Task Update() { + // Arrange: build, load and cache an asset the way the asset manager would var fileSystem = new InMemoryFileSystem().ScopedTo("C:/Test"); - var sut = new HotReloadManager(NullLoggerFactory.Instance, fileSystem); + await fileSystem.WriteAllText(AssetFile, "Hello World"); + + var transcoder = new TextTranscoder(); + var id = new AssetId(AssetFile); + + await AssetEncoder.Encode(id, transcoder, default, fileSystem); + var asset = await AssetDecoder.Decode(id, transcoder, fileSystem); + + using var cache = new AssetPool(); + var live = cache.PutOrLease(id, asset.Value); + + using var sut = new HotReloadManager(NullLoggerFactory.Instance, cache, fileSystem, NoDebounce); + sut.Track(asset, transcoder); + + // Act: change the file the asset was built from + await fileSystem.WriteAllText(AssetFile, "Goodbye World"); + + await Assert.That(() => + { + sut.Update(); + return live.Text; + }) + .Eventually(v => v.IsEqualTo("Goodbye World"), TimeSpan.FromSeconds(5)); + + // Assert: the caller's instance was updated in place, and we left no lease behind + await Assert.That(live.Text).IsEqualTo("Goodbye World"); + cache.Return(id); + } + + [Test] + public async Task Update_RebuildFails() + { + // Arrange: build, load and cache an asset, then make every following rebuild fail + var fileSystem = new InMemoryFileSystem().ScopedTo("C:/Test"); + await fileSystem.WriteAllText(AssetFile, "Hello World"); + + var transcoder = new TranscoderThatCanFail(); + var id = new AssetId(AssetFile); + + await AssetEncoder.Encode(id, transcoder, default, fileSystem); + var asset = await AssetDecoder.Decode(id, transcoder, fileSystem); + + using var cache = new AssetPool(); + var live = cache.PutOrLease(id, asset.Value); + + var sut = new HotReloadManager(NullLoggerFactory.Instance, cache, fileSystem, NoDebounce); + sut.Track(asset, transcoder); + transcoder.ShouldFail = true; + + // Act: one update starts the rebuild, disposing waits for it and finishes it + await fileSystem.WriteAllText(AssetFile, "Goodbye World"); + sut.Update(); + sut.Dispose(); + + // Assert: the rebuild really was attempted, the live asset kept its contents, and returning the last + // lease empties the cache. If the manager leaked its lease the cache throws when it is disposed. + await Assert.That(transcoder.FailedAttempts).IsEqualTo(1); + await Assert.That(live.Text).IsEqualTo("Hello World"); + cache.Return(id); + } +} + + + +internal sealed class TranscoderThatCanFail : TextTranscoder +{ + + public bool ShouldFail { get; set; } = false; + + /// Written on a thread pool thread, only safe to read once the rebuild completed. + public int FailedAttempts { get; private set; } + + public override Task Encode(AssetId id, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer) + { + if (ShouldFail) + { + FailedAttempts++; + throw new InvalidOperationException("Rebuilding this asset failed on purpose"); + } + return base.Encode(id, fileSystem, writer); } } diff --git a/source/CapriKit.Tests/AssetPipeline/HotReloadManagerV3Tests.cs b/source/CapriKit.Tests/AssetPipeline/HotReloadManagerV3Tests.cs deleted file mode 100644 index 7318daa..0000000 --- a/source/CapriKit.Tests/AssetPipeline/HotReloadManagerV3Tests.cs +++ /dev/null @@ -1,123 +0,0 @@ -using CapriKit.AssetPipeline; -using CapriKit.IO; -using CapriKit.IO.Streams; -using Microsoft.Extensions.Logging.Abstractions; -using System.Buffers; - -namespace CapriKit.Tests.AssetPipeline; - -internal class HotReloadManagerV3Tests -{ - private static readonly FilePath AssetFile = new("Hello.txt"); - private static readonly TimeSpan NoDebounce = TimeSpan.Zero; - - [Test] - public async Task Update() - { - // Arrange: build, load and cache an asset the way the asset manager would - var fileSystem = new InMemoryFileSystem().ScopedTo("C:/Test"); - await fileSystem.WriteAllText(AssetFile, "Hello World"); - - var transcoder = new TextTranscoder(); - var id = new AssetId(AssetFile); - - await AssetEncoder.Encode(id, transcoder, default, fileSystem); - var asset = await AssetDecoder.Decode(id, transcoder, fileSystem); - - using var cache = new AssetCache(); - var live = cache.PutOrLease(id, asset.Value); - - using var sut = new HotReloadManagerV3(NullLoggerFactory.Instance, cache, fileSystem, NoDebounce); - sut.Track(asset, transcoder); - - // Act: change the file the asset was built from - await fileSystem.WriteAllText(AssetFile, "Goodbye World"); - - await Assert.That(() => - { - sut.Update(); - return live.Text; - }) - .Eventually(v => v.IsEqualTo("Goodbye World"), TimeSpan.FromSeconds(5)); - - // Assert: the caller's instance was updated in place, and we left no lease behind - await Assert.That(live.Text).IsEqualTo("Goodbye World"); - cache.Return(id); - } - - [Test] - public async Task Update_RebuildFails() - { - // Arrange: build, load and cache an asset, then make every following rebuild fail - var fileSystem = new InMemoryFileSystem().ScopedTo("C:/Test"); - await fileSystem.WriteAllText(AssetFile, "Hello World"); - - var transcoder = new FailingTranscoder(); - var id = new AssetId(AssetFile); - - await AssetEncoder.Encode(id, transcoder, default, fileSystem); - var asset = await AssetDecoder.Decode(id, transcoder, fileSystem); - - using var cache = new AssetCache(); - var live = cache.PutOrLease(id, asset.Value); - - var sut = new HotReloadManagerV3(NullLoggerFactory.Instance, cache, fileSystem, NoDebounce); - sut.Track(asset, transcoder); - transcoder.ShouldFail = true; - - // Act: one update starts the rebuild, disposing waits for it and finishes it - await fileSystem.WriteAllText(AssetFile, "Goodbye World"); - sut.Update(); - sut.Dispose(); - - // Assert: the rebuild really was attempted, the live asset kept its contents, and returning the last - // lease empties the cache. If the manager leaked its lease the cache throws when it is disposed. - await Assert.That(transcoder.FailedAttempts).IsEqualTo(1); - await Assert.That(live.Text).IsEqualTo("Hello World"); - cache.Return(id); - } - - // Hot-swapping needs an asset that can be updated in place, so a mutable holder instead of a plain string - private sealed class TextAsset(string text) - { - public string Text { get; set; } = text; - } - - private class TextTranscoder() : NoSettingsTranscoder(Guid.Parse("{6E4A1D0C-1F73-4C4E-9D2E-0B7F5C6A9E31}"), 1) - { - public override TextAsset Decode(AssetId id, ref SequenceReader reader) - { - return new TextAsset(reader.ReadString()); - } - - public override async Task Encode(AssetId id, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer) - { - var text = await fileSystem.ReadAllText(id.Path); - writer.Write(text); - } - - public override void HotSwap(TextAsset instance, TextAsset newParts) - { - instance.Text = newParts.Text; - } - } - - private sealed class FailingTranscoder : TextTranscoder - { - public bool ShouldFail { get; set; } - - /// Written on a thread pool thread, only safe to read once the rebuild completed. - public int FailedAttempts { get; private set; } - - public override Task Encode(AssetId id, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer) - { - if (ShouldFail) - { - FailedAttempts++; - throw new InvalidOperationException("Rebuilding this asset fails on purpose"); - } - - return base.Encode(id, fileSystem, writer); - } - } -} From f572228ca55600f7ac2f39aedba2f0649041ecd4 Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Thu, 20 Aug 2026 20:56:36 +0200 Subject: [PATCH 45/53] wip --- research/Polling AssetBundleLoader IsReady.md | 238 ++++++++++ .../AssetBundleLoader.cs | 2 +- source/CapriKit.AssetPipeline/AssetManager.cs | 16 +- source/CapriKit.AssetPipeline/AssetPool.cs | 23 +- source/CapriKit.AssetPipeline/README.md | 5 +- .../AssetPipeline/AssetPoolTests.cs | 442 ++++++++++++++++++ 6 files changed, 704 insertions(+), 22 deletions(-) create mode 100644 research/Polling AssetBundleLoader IsReady.md create mode 100644 source/CapriKit.Tests/AssetPipeline/AssetPoolTests.cs diff --git a/research/Polling AssetBundleLoader IsReady.md b/research/Polling AssetBundleLoader IsReady.md new file mode 100644 index 0000000..7696ee7 --- /dev/null +++ b/research/Polling AssetBundleLoader IsReady.md @@ -0,0 +1,238 @@ +# Polling AssetBundleLoader.IsReady + +_Written 2026-08-20, while going through `CapriKit.AssetPipeline` on the `feature/asset_pipeline` branch. We +had just finished the `AssetPool` tests and the fix for `PutOrLease` handing out a disposed asset when the +same instance is put twice. The question that came up next: `AssetBundleLoader.IsReady` is polled +every frame and walks all its handles, and bundles are expected to hold around 40 assets._ + +## What the current code costs + +```csharp +foreach (var handle in handles) +{ + if (!handle.IsResolved) { value = default; return false; } +} +``` + +The loop already stops at the **first** unresolved handle, so it never walks the whole bundle unless the +bundle is nearly done. What it repeats every frame is the prefix of handles that resolved earlier: a +pointer dereference plus a volatile bool read each, over a `List` of references that are +scattered across the heap. + +Worst case per poll is therefore ~40 dependent loads, so tens of nanoseconds (an estimate from the shape of +the loop, not a measurement). At 60 Hz with a handful of bundles in flight that is microseconds per second. + +**So none of the proposals below are a rescue of a hot loop.** Take one because it is simpler than what is there +now, or because it gives you the progress information the `// TODO` in `AssetBundleLoader.cs` asks for. + +## The invariant all three proposals lean on + +`AssetHandle.Resolve` asserts it is only called once and only ever flips `isResolved` from false to true. +A handle that is resolved stays resolved. "Resolved" is monotone, which is what makes it safe to remember +progress between polls without any notification from the handle side. + +## Proposal A: BitArray plus a resume index + +The idea as originally sketched: keep one bit per handle and resume from the first handle that was not +ready last time. Written out, and extended so the bits actually earn their keep by also counting progress: + +```csharp +private readonly BitArray resolved = new(handles.Count); +private int resolvedCount; +private int cursor; // every handle before this one is resolved + +public int Total => handles.Count; +public int Loaded => resolvedCount; + +public bool IsReady([NotNullWhen(true)] out TBundle? value) +{ + if (isReady) { value = result!; return true; } + + // Only look at handles that were not resolved yet, the bits let us skip the ones that were. + for (var i = cursor; i < handles.Count; i++) + { + if (resolved[i] || !handles[i].IsResolved) { continue; } + + resolved[i] = true; + resolvedCount++; + } + + // Everything up to the cursor is resolved, so the next poll can start there. + while (cursor < handles.Count && resolved[cursor]) { cursor++; } + + if (resolvedCount < handles.Count) { value = default; return false; } + + result = factory(new AssetHandleResolver(this)); + isReady = true; + value = result; + return true; +} +``` + +Cost per poll: one bit read per handle that resolved out of order, one volatile read per handle that is +still pending. Both shrink as the bundle loads. Total work over the bundle's life is O(n) plus one pass +over the shrinking tail per poll. + +Notes: +- A `bool[]` is the better container at this size: 40 bytes instead of 8, but no masking and no extra type. + `BitArray` only starts paying off in the thousands of handles. +- Without the counting you can drop the bits entirely, see proposal B for why. +- If you want progress but not the extra arrays, see proposal C. It shrinks the list instead of the scan + range, and it ended up dominating this proposal. + +## Proposal B: resume cursor only (recommended for a yes/no IsReady) + +If `IsReady` stays a yes/no question, one integer replaces the whole thing: + +```csharp +private int cursor; // first handle that was not resolved yet + +public bool IsReady([NotNullWhen(true)] out TBundle? value) +{ + if (isReady) { value = result!; return true; } + + // Handles never go back to unresolved, so everything before the cursor stays resolved and + // each poll only looks at the handles that were still pending during the previous poll. + while (cursor < handles.Count && handles[cursor].IsResolved) { cursor++; } + + if (cursor < handles.Count) { value = default; return false; } + + result = factory(new AssetHandleResolver(this)); + isReady = true; + value = result; + return true; +} +``` + +This is a two line change to the current implementation and it is *less* code than proposal A, not more. + +Cost per poll: one comparison plus one volatile read per handle that resolved since the previous poll. +Summed over the bundle's life that is n reads plus one comparison per poll, which is as good as it gets +without the handles pushing. + +**Why the bits drop out.** Once you also keep a cursor, the loop returns at the first unresolved handle, so +it never looks past it, so it never sets a bit past it. The bits are therefore always "set below the +cursor, clear at and above it", which is exactly what the cursor already says. The BitArray is redundant +unless something makes you keep scanning past the first unresolved handle, and only progress counting does +that. That is the whole difference between A and B. + +**Progress.** `cursor / (float)handles.Count` is a lower bound: monotone and never jumps backwards, so it +drives a progress bar fine, but it under-reports while an early asset is slow. An accurate `37 of 40` +needs proposal C. + +## Proposal C: shrink the pending list + +Instead of remembering where to resume, throw away what is done. The loader keeps its own list of handles +it is still waiting on and swap-removes each one as it resolves: + +```csharp +// A copy: the builder still owns the list it handed us. +private readonly List pending = [.. handles]; +private readonly int total = handles.Count; + +public int Total => total; +public int Loaded => total - pending.Count; +public IReadOnlyList Pending => pending; + +public bool IsReady([NotNullWhen(true)] out TBundle? value) +{ + if (isReady) { value = result!; return true; } + + // Backwards, so that swapping the last handle into the gap cannot skip a handle we still have to see. + for (var i = pending.Count - 1; i >= 0; i--) + { + if (!pending[i].IsResolved) { continue; } + + pending[i] = pending[^1]; + pending.RemoveAt(pending.Count - 1); + } + + if (pending.Count > 0) { value = default; return false; } + + result = factory(new AssetHandleResolver(this)); + isReady = true; + value = result; + return true; +} +``` + +Cost per poll: exactly one volatile read per handle that is still pending, and the scan shrinks as the +bundle loads. That is the price of an exact count. Proposal B can stop at the first unresolved handle +because it only has to answer yes or no, this one has to look at all of them to know how many arrived. + +Notes: +- **The copy is required.** `AssetBundleBuilder.Build` passes its own `Handles` list straight into the + constructor, so compacting it in place would corrupt a second bundle built from the same builder. + (`Build` already re-points every `handle.Owner`, so building twice is not really supported today, but + this proposal should not be the thing that breaks it.) +- Mentioning `handles` only in field initialisers means the primary constructor does not capture it, so + after the copy the loader no longer keeps the builder's list alive. +- It is the only variant that can say *what* it is waiting on: `pending` is exactly the set of assets that + have not arrived, which is what a loading screen or a debug overlay wants to show. +- The swap-remove destroys the order. Nothing reads it today (`AssetHandleResolver` goes through the handle, + not through this list). If that changes, walk forwards and compact with a write index instead, same cost. +- Keeping the resolved handles in a second list gives the same progress number for another allocation and + an `Add` per handle. Nothing in the pipeline consumes them, so I would leave that second list out until + something does. +## Which one + +| | A: bits + cursor | B: cursor | C: shrinking list | +|---|---|---|---| +| Progress | exact count | lower bound | exact count | +| Says what is still pending | no | no | yes | +| Extra state | `bool[]` + count + cursor | cursor | list copy | +| Allocates per bundle | `bool[n]` | nothing | list of n references | +| Reads per poll | one per pending handle, plus a bit per handle that resolved out of order | one while blocked, then one per newly resolved handle | one per pending handle | +| Lines vs. today | ~+12 | ~+2 (and one `foreach` removed) | ~+6 | + +Go with **B** while `IsReady` stays a yes/no question, and with **C** as soon as you want a real progress +number. B is by far the cheapest to poll: while it is blocked on one handle it reads that one handle and +returns, where A and C have to look at everything still pending to keep their count exact. + +**C ended up dominating A**, which is worth writing down since A is where this started. It has fewer moving +parts, it never rescans the resolved-but-out-of-order handles that A's bits exist to skip, and it hands you +the pending set for free. A's only edge is that it leaves the handle list alone, which buys about 320 bytes +per bundle. A stays in this document for the reasoning, not as a candidate. + +Optional add-on, composable with any of them: let `AssetManager` bump a `ResolveGeneration` counter whenever +`Update` resolves anything, and have the loader remember the generation it last saw. A poll on a frame +where nothing resolved at all then returns false after a single int comparison, for every bundle at once. +Costs the loader a reference to the manager, so only worth it if many bundles are in flight. + +## Considered and rejected: let the handles push + +The obvious O(1) answer is a countdown that `AssetHandle.Resolve` decrements through the `Owner` it already +carries: + +```csharp +internal void Resolve(object asset) +{ + value = asset; + isResolved = true; + Owner?.OnHandleResolved(); // remaining-- +} +``` + +This does not survive contact with the loading path, for two reasons: + +1. `AssetManager.Load` resolves straight from the cache while `AssetBundleBuilder.Load` is still running, + which is *before* `AssetBundleBuilder.Build` assigns `handle.Owner`. Every cache hit is a lost + decrement. `Build` would have to count the handles that are already resolved and seed the countdown. +2. That seeding then races: `CreateBundle`, `Load` and `Build` are documented as callable from any thread, + while `Resolve` runs on the main thread inside `Update`. A handle can resolve between `Build` reading + `handle.IsResolved` and `Build` writing `handle.Owner`, which either loses the decrement or counts it + twice. + +Closing that means assigning `Owner` and seeding the count under the `AssetManager.RequestLock`, so the +bundle builder starts depending on the manager's lock and the loading path grows a lock section, all to +save the ~40 bool reads per frame we started with. This is the complexity that was rightly suspected up +front, and the reason all three proposals above stay on the polling side. + +## Related TODOs in the same file + +- `// TODO: how can we put a sort of progress bar and progress information on this thing?` is exactly the fork above: + B if a lower bound that never jumps backwards is enough, C if you want to show a real count. +- `// TODO: Add a method to block and wait without eating all the CPU` is a different problem. None of the + proposals help: blocking needs something to wait on, and the only thing that could signal it is the main + thread inside `AssetManager.Update`, so it would have to be a `ManualResetEventSlim` (or a + `TaskCompletionSource`) that `Update` sets once the bundle's last handle resolved. Worth its own note. diff --git a/source/CapriKit.AssetPipeline/AssetBundleLoader.cs b/source/CapriKit.AssetPipeline/AssetBundleLoader.cs index 7e59e55..14156c9 100644 --- a/source/CapriKit.AssetPipeline/AssetBundleLoader.cs +++ b/source/CapriKit.AssetPipeline/AssetBundleLoader.cs @@ -80,7 +80,7 @@ public bool IsReady([NotNullWhen(true)] out TBundle? value) value = result!; return true; } - + foreach (var handle in handles) { if (!handle.IsResolved) diff --git a/source/CapriKit.AssetPipeline/AssetManager.cs b/source/CapriKit.AssetPipeline/AssetManager.cs index 9406ea4..b64889f 100644 --- a/source/CapriKit.AssetPipeline/AssetManager.cs +++ b/source/CapriKit.AssetPipeline/AssetManager.cs @@ -118,7 +118,7 @@ private async Task RequestAsset(AssetId id, TSettings setting if (build != default && IsUpToDate(transcoder, settings, build, FileSystem)) { var upToDateAsset = await AssetDecoder.Decode(id, transcoder, FileSystem); - Incoming.Write((id, () => MaterializeAsset(upToDateAsset, transcoder))); + Incoming.Write((id, () => TrackAndTakeLease(upToDateAsset, transcoder))); LogLoadedFromFile(Logger, id); } else // If not, try to rebuild and load the asset @@ -130,7 +130,7 @@ private async Task RequestAsset(AssetId id, TSettings setting await AssetEncoder.Encode(id, transcoder, settings, FileSystem); var freshAsset = await AssetDecoder.Decode(id, transcoder, FileSystem); - Incoming.Write((id, () => MaterializeAsset(freshAsset, transcoder))); + Incoming.Write((id, () => TrackAndTakeLease(freshAsset, transcoder))); LogBuildAndLoaded(Logger, id); } } @@ -173,11 +173,11 @@ public void Update() { while (Incoming.TryRead(out var result)) { - var (id, materializer) = result; + var (id, trackAndTakeLease) = result; var handles = Outstanding[id]; foreach (var handle in handles) { - var asset = materializer(); + var asset = trackAndTakeLease(); handle.Resolve(asset); } Outstanding.Remove(id); @@ -237,13 +237,13 @@ private static bool IsUpToDate(IAssetTranscoder - /// Used when an asset is loaded and the handler resolved to register the asset with the cache and hot-reloader. - /// Thread safe: calling this method from multiple threads, even to materialize the same asset is safe. + /// Used when assigning newly loaded assets to their handles. Puts the new asset into the cache, + /// tracks it and then takes a lease for the handle. /// - private TAsset MaterializeAsset(Asset asset, IAssetTranscoder transcoder) + private TAsset TrackAndTakeLease(Asset asset, IAssetTranscoder transcoder) where TAsset : class { - // Though the asset manager does not allow loading the same asset multiple times the cache contract does allow it + // Though the asset manager does not allow loading the same asset multiple times the cache contract does allow it var actualObject = Cache.PutOrLease(asset.Id, asset.Value); var actualWrapper = new Asset(asset.Id, actualObject, asset.BuildMetaData); diff --git a/source/CapriKit.AssetPipeline/AssetPool.cs b/source/CapriKit.AssetPipeline/AssetPool.cs index 60d7dbd..44999e7 100644 --- a/source/CapriKit.AssetPipeline/AssetPool.cs +++ b/source/CapriKit.AssetPipeline/AssetPool.cs @@ -22,10 +22,11 @@ private sealed class Entry(AssetId id, object asset, int refCount) private bool isDisposed; /// - /// Stores the given asset and then leases it. If another caller already stored the asset - /// the is disposed and the stored instance is leased instead. - /// Threading, thread-safe: loading the same asset twice is wasteful but harmless, after calling this - /// method users must stop referencing . + /// Stores the given asset and then leases it. If an asset for the same asset id is added multiple + /// times the first added asset wins. Assets added later just take a lease on the already + /// added one and the candidate is disposed of. + /// Threading: thread-safe, loading the same asset twice is wasteful but harmless, after calling this + /// method users must stop referencing and use the return value of this method instead. /// public TAsset PutOrLease(AssetId id, TAsset candidate) where TAsset : class @@ -36,15 +37,17 @@ public TAsset PutOrLease(AssetId id, TAsset candidate) if (Entries.TryGetValue(id, out var entry)) { - // If someone tries to add the same instance twice - // just return it, do not schedule it for dispose - if (object.ReferenceEquals(candidate, entry)) + // If someone adds the same asset twice we ensure that + // the version already in the cache wins and update + // the ref count accordingly + if (!object.ReferenceEquals(candidate, entry.Asset)) { - return candidate; + // If a different instance was added for the same id + // the candidate also needs to be disposed. + PendingDispose.Enqueue(new Entry(id, candidate, 0)); } - PendingDispose.Enqueue(new Entry(id, candidate, 0)); - + var asset = Cast(entry, id); entry.RefCount++; return asset; diff --git a/source/CapriKit.AssetPipeline/README.md b/source/CapriKit.AssetPipeline/README.md index 349ca2e..f66bd37 100644 --- a/source/CapriKit.AssetPipeline/README.md +++ b/source/CapriKit.AssetPipeline/README.md @@ -1,10 +1,10 @@ # CapriKit.AssetPipeline -Pluggable asset pipeline that provides multi-threaded building and loading and hot-reloading of assets. +Reusable asset pipeline that provides multi-threaded building, loading and hot-reloading of assets. ## Usage -1. Create the asset manager and register a transcoder, a class that knows how to build and load one asset type: +1. Create the asset manager and register a transcoder, an implementation of `IAssetTranscoder` that knows how to build and load one asset type: ``` var assetManager = new AssetManager(LoggerFactory, ScopedFileSystem); assetManager.RegisterTranscoder(new VertexShaderTranscoder(GraphicsDevice)); @@ -48,4 +48,3 @@ assetManager.Unload(bundle); Hot reloading happens automatically if the files that were used to build the asset are present and change. ## Implementation -The AssetPool ensures that files diff --git a/source/CapriKit.Tests/AssetPipeline/AssetPoolTests.cs b/source/CapriKit.Tests/AssetPipeline/AssetPoolTests.cs new file mode 100644 index 0000000..0ef152f --- /dev/null +++ b/source/CapriKit.Tests/AssetPipeline/AssetPoolTests.cs @@ -0,0 +1,442 @@ +using CapriKit.AssetPipeline; + +namespace CapriKit.Tests.AssetPipeline; + +/// +/// The pool is where the loading threads, the hot-reloader and the main thread meet. A mistake in its +/// reference counting either disposes an asset that is still in use or leaks it, so these tests go beyond +/// the happy path and pin down the lease/return contract, the deferred disposal and the threading. +/// +internal class AssetPoolTests +{ + private static readonly AssetId Id = new("Hello.txt"); + private static readonly AssetId OtherId = new("Goodbye.txt"); + + [Test] + public async Task PutOrLease() + { + var pool = new AssetPool(); + var resource = new Resource(); + + var stored = pool.PutOrLease(Id, resource); + + await Assert.That(stored).IsSameReferenceAs(resource); + await Assert.That(resource.DisposeCount).IsEqualTo(0); + + // Putting an asset takes the first lease on it, returning that lease releases the asset again + pool.Return(Id); + pool.DisposeReleased(); + await Assert.That(resource.DisposeCount).IsEqualTo(1); + + pool.Dispose(); + } + + [Test] + public async Task PutOrLease_DisposesTheLoserAndLeasesTheWinner() + { + var pool = new AssetPool(); + var winner = new Resource(); + var loser = new Resource(); + + pool.PutOrLease(Id, winner); + var stored = pool.PutOrLease(Id, loser); + + await Assert.That(stored).IsSameReferenceAs(winner); + + // Disposal always waits for the main thread, never happens inside PutOrLease itself + await Assert.That(loser.DisposeCount).IsEqualTo(0); + pool.DisposeReleased(); + await Assert.That(loser.DisposeCount).IsEqualTo(1); + await Assert.That(winner.DisposeCount).IsEqualTo(0); + + // Both calls took a lease on the winner, so both have to be returned + pool.Return(Id); + pool.DisposeReleased(); + await Assert.That(winner.DisposeCount).IsEqualTo(0); + + pool.Return(Id); + pool.DisposeReleased(); + await Assert.That(winner.DisposeCount).IsEqualTo(1); + + pool.Dispose(); + } + + [Test] + public async Task PutOrLease_LeasesWithoutDisposingWhenTheSameInstanceIsPutTwice() + { + // The asset manager materializes an asset once per waiting handle but always from the same loaded + // instance, so putting the identical object twice is a normal path rather than a corner case. + var pool = new AssetPool(); + var resource = new Resource(); + + pool.PutOrLease(Id, resource); + var stored = pool.PutOrLease(Id, resource); + + await Assert.That(stored).IsSameReferenceAs(resource); + + // The instance is live and now leased twice, collecting must not touch it + pool.DisposeReleased(); + await Assert.That(resource.DisposeCount).IsEqualTo(0); + + pool.Return(Id); + pool.DisposeReleased(); + await Assert.That(resource.DisposeCount).IsEqualTo(0); + + pool.Return(Id); + pool.DisposeReleased(); + await Assert.That(resource.DisposeCount).IsEqualTo(1); + + pool.Dispose(); + } + + [Test] + public async Task PutOrLease_ThrowsWhenTheAssetIsStoredAsAnotherType() + { + var pool = new AssetPool(); + var stored = new Resource(); + var wrongType = new OtherResource(); + + pool.PutOrLease(Id, stored); + + await Assert.That(() => pool.PutOrLease(Id, wrongType)).Throws(); + + // The rejected candidate is still cleaned up, and the failed call did not take a lease + pool.DisposeReleased(); + await Assert.That(wrongType.DisposeCount).IsEqualTo(1); + await Assert.That(stored.DisposeCount).IsEqualTo(0); + + pool.Return(Id); + pool.DisposeReleased(); + await Assert.That(stored.DisposeCount).IsEqualTo(1); + + pool.Dispose(); + } + + [Test] + public async Task TryLease() + { + var pool = new AssetPool(); + var resource = new Resource(); + pool.PutOrLease(Id, resource); + + var found = pool.TryLease(Id, out var leased); + + await Assert.That(found).IsTrue(); + await Assert.That(leased).IsSameReferenceAs(resource); + + pool.Return(Id); + pool.Return(Id); + pool.DisposeReleased(); + pool.Dispose(); + } + + [Test] + public async Task TryLease_ReturnsFalseForAnUnknownAsset() + { + var pool = new AssetPool(); + + var found = pool.TryLease(Id, out var leased); + + await Assert.That(found).IsFalse(); + await Assert.That(leased).IsNull(); + + pool.Dispose(); + } + + [Test] + public async Task TryLease_ReturnsFalseForAnAssetThatIsWaitingToBeDisposed() + { + // Returning the last lease evicts the asset immediately and only defers the disposal itself, so a + // late lease has to miss instead of handing out an asset that is about to be disposed. + var pool = new AssetPool(); + var resource = new Resource(); + pool.PutOrLease(Id, resource); + pool.Return(Id); + + var found = pool.TryLease(Id, out var leased); + + await Assert.That(found).IsFalse(); + await Assert.That(leased).IsNull(); + + pool.DisposeReleased(); + await Assert.That(resource.DisposeCount).IsEqualTo(1); + + pool.Dispose(); + } + + [Test] + public async Task TryLease_ThrowsWhenTheAssetIsStoredAsAnotherType() + { + var pool = new AssetPool(); + var resource = new Resource(); + pool.PutOrLease(Id, resource); + + await Assert.That(() => pool.TryLease(Id, out _)).Throws(); + + // The failed lease must not have counted, so a single return still releases the asset + pool.Return(Id); + pool.DisposeReleased(); + await Assert.That(resource.DisposeCount).IsEqualTo(1); + + pool.Dispose(); + } + + [Test] + public async Task Return() + { + var pool = new AssetPool(); + var resource = new Resource(); + pool.PutOrLease(Id, resource); + pool.TryLease(Id, out _); + + pool.Return(Id); + + // One lease is left, so the asset stays alive and leaseable + pool.DisposeReleased(); + await Assert.That(resource.DisposeCount).IsEqualTo(0); + await Assert.That(pool.TryLease(Id, out _)).IsTrue(); + pool.Return(Id); + + pool.Return(Id); + + // The last return evicts the asset but leaves the disposal to the main thread + await Assert.That(resource.DisposeCount).IsEqualTo(0); + pool.DisposeReleased(); + await Assert.That(resource.DisposeCount).IsEqualTo(1); + + pool.Dispose(); + } + + [Test] + public async Task Return_ThrowsForAnAssetThatWasNeverStored() + { + var pool = new AssetPool(); + + await Assert.That(() => pool.Return(Id)).Throws(); + + pool.Dispose(); + } + + [Test] + public async Task Return_ThrowsWhenReturnedMoreOftenThanLeased() + { + var pool = new AssetPool(); + pool.PutOrLease(Id, new Resource()); + pool.Return(Id); + + // The entry is gone once the last lease came back, so an extra return is a bug in the caller + await Assert.That(() => pool.Return(Id)).Throws(); + + pool.DisposeReleased(); + pool.Dispose(); + } + + [Test] + public async Task DisposeReleased() + { + var pool = new AssetPool(); + var first = new Resource(); + var second = new Resource(); + pool.PutOrLease(Id, first); + pool.PutOrLease(OtherId, second); + pool.Return(Id); + pool.Return(OtherId); + + pool.DisposeReleased(); + + await Assert.That(first.DisposeCount).IsEqualTo(1); + await Assert.That(second.DisposeCount).IsEqualTo(1); + + // Collecting again must not dispose anything a second time + pool.DisposeReleased(); + await Assert.That(first.DisposeCount).IsEqualTo(1); + await Assert.That(second.DisposeCount).IsEqualTo(1); + + pool.Dispose(); + } + + [Test] + public async Task DisposeReleased_IgnoresAssetsThatAreNotDisposable() + { + var pool = new AssetPool(); + pool.PutOrLease(Id, new PlainResource()); + pool.Return(Id); + + pool.DisposeReleased(); + + await Assert.That(pool.TryLease(Id, out _)).IsFalse(); + + pool.Dispose(); + } + + [Test] + public async Task DisposeReleased_DoesNotHoldTheLockWhileDisposing() + { + // Assets dispose graphics resources and may well touch the pool themselves. Doing that while + // holding the lock would deadlock against every other thread that loads or returns an asset. + var pool = new AssetPool(); + pool.PutOrLease(OtherId, new Resource()); + + var otherThreadCouldUseThePool = false; + var reentrant = new CallbackResource(() => + { + var lease = Task.Run(() => pool.TryLease(OtherId, out _)); + otherThreadCouldUseThePool = lease.Wait(TimeSpan.FromSeconds(5)) && lease.Result; + }); + + pool.PutOrLease(Id, reentrant); + pool.Return(Id); + + pool.DisposeReleased(); + + await Assert.That(otherThreadCouldUseThePool).IsTrue(); + + pool.Return(OtherId); + pool.Return(OtherId); + pool.DisposeReleased(); + pool.Dispose(); + } + + [Test] + public async Task Dispose() + { + var pool = new AssetPool(); + var resource = new Resource(); + pool.PutOrLease(Id, resource); + pool.Return(Id); + + // Everything came back, so disposing is clean and still collects what was queued + pool.Dispose(); + + await Assert.That(resource.DisposeCount).IsEqualTo(1); + } + + [Test] + public async Task Dispose_ThrowsWhenAssetsWereNotReturned() + { + var pool = new AssetPool(); + var resource = new Resource(); + pool.PutOrLease(Id, resource); + + // The pool is the only place that can catch a bundle that was never unloaded + await Assert.That(() => pool.Dispose()).Throws(); + + // The leaked asset is reported, not disposed: somebody out there still holds a reference to it + await Assert.That(resource.DisposeCount).IsEqualTo(0); + } + + [Test] + public async Task Dispose_OnlyReportsLeaksOnce() + { + var pool = new AssetPool(); + pool.PutOrLease(Id, new Resource()); + + await Assert.That(() => pool.Dispose()).Throws(); + + // A pool in a using block is disposed a second time while the first exception unwinds + await Assert.That(() => pool.Dispose()).ThrowsNothing(); + } + + [Test] + public async Task Dispose_MakesThePoolUnusable() + { + var pool = new AssetPool(); + pool.Dispose(); + + await Assert.That(() => pool.PutOrLease(Id, new Resource())).Throws(); + await Assert.That(() => pool.TryLease(Id, out _)).Throws(); + await Assert.That(() => pool.Return(Id)).Throws(); + + // Collecting stays safe so that a game loop that is shutting down does not have to check + await Assert.That(() => pool.DisposeReleased()).ThrowsNothing(); + } + + [Test] + public async Task PutOrLease_IsThreadSafe() + { + // Several loading threads can finish the same asset at the same time. Exactly one instance may + // win, every caller must get a lease on that winner and every loser must be cleaned up. + const int threads = 16; + var pool = new AssetPool(); + var candidates = Enumerable.Range(0, threads).Select(_ => new Resource()).ToArray(); + + var results = await Task.WhenAll(candidates.Select(candidate => Task.Run(() => pool.PutOrLease(Id, candidate)))); + + var winner = results[0]; + await Assert.That(results.Distinct().Count()).IsEqualTo(1); + await Assert.That(candidates.Count(candidate => ReferenceEquals(candidate, winner))).IsEqualTo(1); + + pool.DisposeReleased(); + await Assert.That(winner.DisposeCount).IsEqualTo(0); + await Assert.That(candidates.Sum(candidate => candidate.DisposeCount)).IsEqualTo(threads - 1); + + // Every call took a lease, so the winner only dies after the last one comes back + for (var i = 0; i < threads; i++) + { + pool.Return(Id); + } + + pool.DisposeReleased(); + await Assert.That(winner.DisposeCount).IsEqualTo(1); + + pool.Dispose(); + } + + [Test] + public async Task TryLease_IsThreadSafe() + { + // The reference count is the only thing keeping an asset alive, so it has to survive many threads + // leasing and returning the same asset at the same time. + const int threads = 8; + const int iterations = 500; + + var pool = new AssetPool(); + var resource = new Resource(); + + // This lease keeps the asset alive for the entire test + pool.PutOrLease(Id, resource); + + await Task.WhenAll(Enumerable.Range(0, threads).Select(thread => Task.Run(() => + { + for (var i = 0; i < iterations; i++) + { + if (pool.TryLease(Id, out _)) + { + pool.Return(Id); + } + } + }))); + + // The asset was never evicted along the way, so only the initial lease is left + pool.DisposeReleased(); + await Assert.That(resource.DisposeCount).IsEqualTo(0); + + pool.Return(Id); + pool.DisposeReleased(); + await Assert.That(resource.DisposeCount).IsEqualTo(1); + + pool.Dispose(); + } + + /// Counts disposals so that tests can tell a missing, a late and a double dispose apart. + private class CountingResource : IDisposable + { + private int disposeCount; + + public int DisposeCount => Volatile.Read(ref disposeCount); + + public void Dispose() => Interlocked.Increment(ref disposeCount); + } + + private sealed class Resource : CountingResource; + + /// Unrelated to so that casting one to the other fails. + private sealed class OtherResource : CountingResource; + + private sealed class PlainResource; + + private sealed class CallbackResource(Action onDispose) : IDisposable + { + public void Dispose() => onDispose(); + } +} From ac948154b20ced9f3d2e0f4559dfa6c95aebf60f Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Thu, 20 Aug 2026 23:31:57 +0200 Subject: [PATCH 46/53] wio2 --- source/CapriKit.AssetPipeline/Asset.cs | 13 +++- .../AssetBundleLoader.cs | 65 +++++++++++++------ 2 files changed, 58 insertions(+), 20 deletions(-) diff --git a/source/CapriKit.AssetPipeline/Asset.cs b/source/CapriKit.AssetPipeline/Asset.cs index 4756f3f..23624af 100644 --- a/source/CapriKit.AssetPipeline/Asset.cs +++ b/source/CapriKit.AssetPipeline/Asset.cs @@ -7,7 +7,18 @@ namespace CapriKit.AssetPipeline; /// /// Virtual file path that points to the file the asset originates from. /// Optional key to a sub-resources in Path. -public record AssetId(FilePath Path, string Key = ""); +public record AssetId(FilePath Path, string Key = "") +{ + public override string ToString() + { + if (string.IsNullOrEmpty(Key)) + { + return Path; + } + + return $"{Path}:{Key}"; + } +} /// /// An asset, including its identifier and information on how it was built. diff --git a/source/CapriKit.AssetPipeline/AssetBundleLoader.cs b/source/CapriKit.AssetPipeline/AssetBundleLoader.cs index 14156c9..2a8e498 100644 --- a/source/CapriKit.AssetPipeline/AssetBundleLoader.cs +++ b/source/CapriKit.AssetPipeline/AssetBundleLoader.cs @@ -68,36 +68,63 @@ public sealed class AssetBundleLoader(Func Pending = [.. handles]; + + /// + /// The number of assets in this bundle. + /// + public int Total { get; } = handles.Count; + + /// + /// The number of assets that have completed loading, updated every time is called. + /// + public int Loaded => Total - Pending.Count; + + /// + /// The latest item that completed loading, updated every time is called. + /// + public AssetId? LastCompletedItem { get; private set; } + private TBundle? result; + private bool isReady; - // Single threaded! - // TODO: can we reduce the number of things we need to check each frame? + /// + /// Checks whether the bundle finished loading, and if so builds returns it. + /// Threading: primary thread only. + /// public bool IsReady([NotNullWhen(true)] out TBundle? value) { - if (isReady) - { - value = result!; - return true; - } - - foreach (var handle in handles) + if (isReady) { value = result!; return true; } + + for (var i = Pending.Count - 1; i >= 0; i--) { - if (!handle.IsResolved) + var handle = Pending[i]; + if (handle.IsResolved) { - value = default; - return false; + LastCompletedItem = handle.Id; + Pending[i] = Pending[^1]; + Pending.RemoveAt(Pending.Count - 1); } } - result = factory(new AssetHandleResolver(this)); - isReady = true; + if (Pending.Count > 0) { value = default; return false; } - value = result; + result = value = factory(new AssetHandleResolver(this)); + isReady = true; return true; } - // TODO: Add a method to block and wait without eating all the CPU. - - // TODO: how can we put a sort of progress bar and progress information on this thing? + /// + /// Busy waits until the bundle finishes loading, then builds and returns it. + /// Threading: primary thread only. + /// + public TBundle WaitUntilReady() + { + var wait = new SpinWait(); + while (true) + { + if (IsReady(out var bundle)) { return bundle; } + wait.SpinOnce(); + } + } } From 50ffd0c2734db2c1c684ad6ebbcedf65088f0f4a Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Tue, 25 Aug 2026 20:38:25 +0200 Subject: [PATCH 47/53] wip --- .../Recovering from a failed asset load.md | 263 ++++++++++++++++++ source/CapriKit.AssetPipeline/AssetManager.cs | 2 +- .../AssetPipeline/AssetManagerTests.cs | 99 +++++++ .../AssetPipeline/HotReloadManagerTests.cs | 20 +- 4 files changed, 379 insertions(+), 5 deletions(-) create mode 100644 research/Recovering from a failed asset load.md diff --git a/research/Recovering from a failed asset load.md b/research/Recovering from a failed asset load.md new file mode 100644 index 0000000..82003f7 --- /dev/null +++ b/research/Recovering from a failed asset load.md @@ -0,0 +1,263 @@ +# Recovering from a failed asset load + +_Written 2026-08-25, on the `feature/asset_pipeline` branch. We had just added +`AssetManagerTests.Load_RetriesAnAssetWhoseFirstLoadFailed`, which fails on purpose, to keep this bug from +being forgotten. This note is the plan for fixing it properly._ + +## What is actually broken + +Three separate defects, all in the same handful of lines. Only the first one is the wedge, but a fix that +does not address the other two just moves the problem. + +**1. The request is never cleared.** `AssetManager.Update` only removes the `Outstanding` entry on the +success path: + +```csharp +while (Incoming.TryRead(out var result)) +{ + var (id, trackAndTakeLease) = result; + var handles = Outstanding[id]; + // snip: resolve every handle + Outstanding.Remove(id); // never reached when the request failed +} +``` + +A failure travels through a *different* route: `LightweightChannel.Write(ExceptionDispatchInfo)` puts it in +a second queue, and `TryRead` drains that queue first and throws. So `Update` throws before it can clean +anything up, `Outstanding[id]` survives, and because `Load` hands every later request for that id to the +existing list instead of starting a new one, the asset can never be loaded again. + +**2. The error does not know which asset it belongs to.** The exception queue in `LightweightChannel` is +untyped and separate from the item queue, so by the time `Update` catches it there is no id to clean up +even if we wanted to. This is why the fix cannot be local to `Update` as it stands. + +**3. A failure takes down more than the asset.** `Update` throws from inside the lock, which skips +`Cache.DisposeReleased()` and `HotReloadManager.Update()` for that frame, and abandons any *successful* +loads still sitting in the queue behind the failure. + +## The half that is not a choice + +Whatever policy we pick, the cleanup has to happen **on the main thread, inside the same `RequestLock` +section that `Load` uses**, and the failure has to carry its id to get there. Route both outcomes through +the one queue: + +```csharp +// One item per finished request, successful or not, so Update sees both the same way. +internal readonly record struct LoadResult(AssetId Id, Func? Materialize, ExceptionDispatchInfo? Error); +``` + +`Load`'s failure handler stops using the channel's exception queue: + +```csharp +Task.Run(() => RequestAsset(id, settings)).FireAndForget( + ex => + { + LogFailed(Logger, id); + Incoming.Write(new LoadResult(id, null, ex)); // was: Incoming.Write(ex) + }); +``` + +and `Update` clears the entry for both outcomes: + +```csharp +lock (RequestLock) +{ + while (Incoming.TryRead(out var result)) + { + // Removing here is the whole fix: a handle either made it into this list before we took the lock, + // or it misses the list entirely and its Load starts a fresh request. + if (!Outstanding.Remove(result.Id, out var handles)) { continue; } + + if (result.Error is not null) { /* policy, see below */ continue; } + + foreach (var handle in handles) { handle.Resolve(result.Materialize!()); } + } +} +``` + +After this, `LightweightChannel.Write(ExceptionDispatchInfo)` is unused by the asset pipeline. Leave the +overload alone (it is a general-purpose primitive), but know that this path no longer relies on it. + +## The half that is a choice: what the app sees + +All three options below build on that same core. **A and B both give you the "proper exception that quits +the app if uncaught" behaviour** you asked for; they differ in *where* it is thrown. + +### Option A: `Update` throws + +```csharp +if (result.Error is not null) +{ + // The entry is already gone, so a later Load starts a clean request even if somebody catches this. + throw new AssetLoadException(result.Id, result.Error.SourceException); +} +``` + +Smallest possible diff on top of the core: no changes to `AssetHandle` or `AssetBundleLoader`. The handles +of the failed request are simply left unresolved forever. + +That last part is the catch. It is fine while the exception is uncaught and the process dies, but an app +that catches it is left with a bundle that never completes and no way to find out why — a silent hang. +Option A is therefore *only* correct as a fail-fast policy; catching it is unsupported. It also keeps +defect 3: the throw still skips `DisposeReleased` and the hot-reload update for that frame. + +### Option B: the handle carries the failure, `IsReady` throws (recommended) + +Give `AssetHandle` an error next to its value. Writing `error` before the `volatile` `isResolved` gives the +same release ordering the existing `value` write relies on: + +```csharp +private Exception? error; +internal Exception? Error => error; + +internal void Fail(Exception ex) +{ + Debug.Assert(isResolved == false); + error = ex; + isResolved = true; // volatile write publishes error too +} +``` + +`Update` fails the handles instead of throwing, so it stays a pure bookkeeping call: + +```csharp +if (result.Error is not null) +{ + foreach (var handle in handles) { handle.Fail(result.Error.SourceException); } + continue; +} +``` + +and `AssetBundleLoader.IsReady` treats a failed handle as arrived, then throws once everything is +in: + +```csharp +for (var i = Pending.Count - 1; i >= 0; i--) +{ + var handle = Pending[i]; + if (!handle.IsResolved) { continue; } + if (handle.Error is not null) { (failures ??= []).Add(new AssetLoadException(handle.Id, handle.Error)); } + // snip: existing swap-remove +} + +if (Pending.Count > 0) { value = default; return false; } + +// Everything arrived but some of it failed. Throwing here puts the error in front of the code that +// actually wanted the asset, rather than in the middle of the manager's bookkeeping. +if (failures is not null) { throw failures.Count == 1 ? failures[0] : new AggregateException(failures); } +``` + +`IsReady` is called from the game loop, so an uncaught `AssetLoadException` still takes the process down — +your requirement is met. What you gain over A is that `Update` never throws, so the cache collection and +the hot-reload pass always run, and the failure is attributed to a specific bundle and asset. + +Note that `IsReady` will throw again on every subsequent call, since `Pending` is empty but `isReady` was +never set. That is the right idempotent behaviour, but it means a caught exception throws once per frame. + +### Option C: the loader reports, nothing throws + +Same `AssetHandle.Fail` as B, but the loader exposes the failure instead of throwing: + +```csharp +public bool IsFaulted => Failures.Count > 0; +public IReadOnlyList Failures => failures ?? []; +``` + +`IsReady` returns false forever while faulted. This is what you want the day an app wants to substitute a +placeholder asset and carry on. It is the wrong *default*: an app that forgets to check `IsFaulted` sits on +a loading screen forever with nothing but a log line to show for it. + +Worth building **on top of** B rather than instead of it — B can grow `IsFaulted` and a non-throwing +`TryGetResult` later without changing what the throwing path does. + +## The load / hot-reload asymmetry + +Good news: the hot-reload half already behaves the way you want, and no option above changes it. +`HotReloadManager.FinishReload` catches everything, logs it, keeps the live asset's contents and returns +the lease in a `finally`; the faulted task's `.Exception` is read by `LogReloadFailed`, which marks it +observed so it never reaches `TaskScheduler.UnobservedTaskException`. Hot reload does not use `Incoming` at +all, so failures there cannot leak into the load path. + +What the fix should add is the *statement* of that split, because the same transcoder bug now produces a +hard crash in one path and a log line in the other, and that looks arbitrary until you say why: + +- **Load**: there is no valid asset to fall back on, so the app cannot sensibly continue. Fail fast. +- **Hot reload**: there is a perfectly good asset already live, and this is a development-time convenience. + Never take down the game over it. + +Concretely: document that contract on `IAssetTranscoder.Encode` and `.Decode`, and add a +regression test that a hot-reload failure does not throw out of `AssetManager.Update()` (today +`HotReloadManagerTests.Update_RebuildFails` only exercises `HotReloadManager` directly, never through the +manager). Under option A that test is load-bearing, because `Update` becomes a method that sometimes throws. + +## Side cleanup worth folding in + +`RequestAsset` calls `GetTranscoder()` on the thread pool, so a missing or mismatched +transcoder — a programmer error, not an asset failure — arrives through the same failure path as a corrupt +file. Hoisting that call into `Load` makes it throw synchronously, on the calling thread, with a clean +stack, before any `Outstanding` entry exists. Small and independent of the options above. + +## Open question: retrying a permanently broken asset + +Once the wedge is gone, every `Load` of a broken asset starts a fresh build. In practice `Load` is called +once per bundle rather than once per frame, so this is unlikely to become a rebuild storm — but nothing +stops it either. If it ever bites, the cheap answer is to remember failed ids with a timestamp and refuse to +retry within a few seconds, or to require an explicit `assetManager.Forget(id)`. Not worth building yet. + +## Which one + +| | A: `Update` throws | B: `IsReady` throws | C: loader reports | +|---|---|---|---| +| Wedge fixed | yes | yes | yes | +| Kills the app if uncaught | yes | yes | no | +| Error names the bundle and asset | id only | yes | yes | +| `Update` stays non-throwing | no | yes | yes | +| Cache collect + hot reload still run that frame | no | yes | yes | +| App can recover deliberately | no (silent hang) | no | yes | +| Touches | `Update` | `Update`, `AssetHandle`, `AssetBundleLoader` | B, plus loader API | + +Go with **B**. It gives you the fail-fast crash you asked for, but throws it where the app asked for the +asset instead of in the middle of the manager's bookkeeping, which keeps defect 3 fixed as well. A is worth +knowing about as the two-line version if you want the wedge gone today and the rest later. C is the natural +follow-up once something actually wants to survive a missing asset. + +## Test impact + +`Load_RetriesAnAssetWhoseFirstLoadFailed` goes green on `Attempts == 3` under all three options. Its last +assertion is the one that changes: + +- **A**: unchanged — the failed bundle's handles stay unresolved, so `IsReady` is still `false`. +- **B**: becomes `await Assert.That(() => failedLoader.IsReady(out _)).Throws();` +- **C**: `IsReady` stays `false` and `failedLoader.IsFaulted` is `true`. + +Two tests worth adding while in here: + +1. A hot-reload failure does not throw out of `AssetManager.Update()` (the asymmetry above). +2. Two concurrent `Load` calls for the same failing id both get failed, and neither is silently dropped — + this is the race that the naive fix gets wrong, see below. + +## Considered and rejected: clearing `Outstanding` from the thread pool + +The obvious two-line fix, and the one I probed with while checking that the new test can go green: + +```csharp +ex => +{ + LogFailed(Logger, id); + lock (RequestLock) { Outstanding.Remove(id); } + Incoming.Write(ex); +}); +``` + +It does turn the suite green, which is exactly why it is worth writing down as a trap. It races: + +1. Thread A calls `Load(id)`, creates `Outstanding[id] = [h1]` and starts the request. +2. The request fails; the continuation queues up on `RequestLock`. +3. Thread B calls `Load(id)`, wins the lock, sees the entry is still there and adds `h2` to it. B believes + its load is in flight. +4. The continuation takes the lock and removes the entry — dropping **both** `h1` and `h2`. + +`h2` now never resolves and no request was ever started for it, so the wedge has been traded for a silently +dropped handle, which is harder to notice. Doing the removal on the main thread inside `Update`, in the +same lock section that materializes successes, is what closes this: `h2` is either in the list we are about +to fail, or it missed the list and its own `Load` started a fresh request. There is no third case. diff --git a/source/CapriKit.AssetPipeline/AssetManager.cs b/source/CapriKit.AssetPipeline/AssetManager.cs index b64889f..1720fd5 100644 --- a/source/CapriKit.AssetPipeline/AssetManager.cs +++ b/source/CapriKit.AssetPipeline/AssetManager.cs @@ -140,7 +140,7 @@ private async Task RequestAsset(AssetId id, TSettings setting /// Threading: Unload updates the internal state of the bundle using a lock so that it is safe /// to unload the same bundle from multiple threads. /// - public void Unload(AssetBundleLoader bundle) + public void Unload(AssetBundleLoader bundle) // TODO: this should unload an assetbundle, not a loader, but that is not a formal class/interface yet and I don't want it to become more complicated to define an asset bundle. What to do? { try { diff --git a/source/CapriKit.Tests/AssetPipeline/AssetManagerTests.cs b/source/CapriKit.Tests/AssetPipeline/AssetManagerTests.cs index e4c5937..0687496 100644 --- a/source/CapriKit.Tests/AssetPipeline/AssetManagerTests.cs +++ b/source/CapriKit.Tests/AssetPipeline/AssetManagerTests.cs @@ -11,6 +11,7 @@ internal class AssetManagerTests { private DirectoryPath? WorkingDirectory; private readonly FilePath AssetFile = new("Hello.txt"); + private readonly FilePath HealthyFile = new("Goodbye.txt"); private const string TranscoderText = "Hello World"; [Before(Test)] @@ -78,6 +79,104 @@ await Assert.That(() => // TODO: test the dispose path. Check the errors if assets are not returned and that returning both bundles correctly disposes everything. } + + /// + /// FAILS ON PURPOSE: it asserts the behaviour we want, which the AssetManager does not have yet. + /// + /// A load that fails wedges its asset id for the rest of the manager's life. RequestAsset reports the + /// failure through the incoming channel, but only its success path writes a materializer, so Update + /// never reaches the `Outstanding.Remove(id)` that clears the request. The dead handle list survives, + /// and because Load hands every later request for that id to the existing list instead of starting a + /// new one, the asset can never be loaded again, not even after the cause of the failure is gone. + /// + /// The fix is to clear the Outstanding entry when a request fails, next to the `Incoming.Write(ex)` + /// in AssetManager.Load, so that the next Load starts a fresh request. + /// + [Test] + public async Task Load_RetriesAnAssetWhoseFirstLoadFailed() + { + var fileSystem = new InMemoryFileSystem().ScopedTo("C:/Test"); + await fileSystem.WriteAllText(AssetFile, TranscoderText); + + var transcoder = new TranscoderThatCanFail { ShouldFail = true }; + + // Deliberately not a `using`: AssetPool.Dispose throws when leases are outstanding, and an exception + // from a dispose during unwinding replaces the assertion that actually failed. Disposing at the end + // keeps the leak check but lets a real failure report itself. + var assetManager = new AssetManager(NullLoggerFactory.Instance, fileSystem); + assetManager.RegisterTranscoder(transcoder); + + var id = new AssetId(AssetFile); + + // Act: the first load fails while building and Update rethrows that failure on the main thread + var failedBuilder = assetManager.CreateBundle(); + var failedHandle = failedBuilder.Load(id); + var failedLoader = failedBuilder.Build(resolver => new TestBundle(resolver.Get(failedHandle))); + + Exception? failure = null; + await Assert.That(() => + { + try { assetManager.Update(); } + catch (Exception ex) { failure = ex; } + return failure is not null; + }) + .Eventually(v => v.IsTrue(), TimeSpan.FromSeconds(5)); + + await Assert.That(failure).IsTypeOf(); + await Assert.That(transcoder.Attempts).IsEqualTo(1); + + // Act: take away the reason the build failed and ask for the very same asset again + transcoder.ShouldFail = false; + + var retryBuilder = assetManager.CreateBundle(); + var retryHandle = retryBuilder.Load(id); + var retryLoader = retryBuilder.Build(resolver => new TestBundle(resolver.Get(retryHandle))); + + // Control: a second, untouched asset requested at the same moment. Waiting for it proves that a + // failure does not stop the manager as a whole, so a red test really is about this one asset id. + await fileSystem.WriteAllText(HealthyFile, TranscoderText); + + var healthyBuilder = assetManager.CreateBundle(); + var healthyHandle = healthyBuilder.Load(new AssetId(HealthyFile)); + var healthyLoader = healthyBuilder.Build(resolver => new TestBundle(resolver.Get(healthyHandle))); + + await Assert.That(() => + { + assetManager.Update(); + return healthyLoader.IsReady(out _); + }) + .Eventually(v => v.IsTrue(), TimeSpan.FromSeconds(5)); + + // Pump until the retry has built, and stop early once it has so that the green path stays quick. + // Today it never happens and we spend the whole window, which is what makes the failure below crisp + // instead of a five second timeout somewhere else. + for (var i = 0; i < 200 && transcoder.Attempts < 3; i++) + { + assetManager.Update(); + await Task.Delay(10); + } + + // Assert: the failure, the control and the retry each reached the transcoder exactly once. + // THIS IS THE ASSERTION THAT FAILS TODAY: the retry never starts, so it stops at 2. + await Assert.That(transcoder.Attempts).IsEqualTo(3); + + await Assert.That(() => + { + assetManager.Update(); + return retryLoader.IsReady(out _); + }) + .Eventually(v => v.IsTrue(), TimeSpan.FromSeconds(5)); + + // The first bundle stays unresolved even after the fix. Its exception already surfaced out of + // Update and a handle carries no way to report one, so those handles are simply dead. + await Assert.That(failedLoader.IsReady(out _)).IsFalse(); + + // Only the two resolved bundles hold a lease, so a clean dispose also proves that the failed + // request never leaked one of its own. + assetManager.Unload(retryLoader); + assetManager.Unload(healthyLoader); + assetManager.Dispose(); + } } internal record TestBundle(TextAsset Asset); diff --git a/source/CapriKit.Tests/AssetPipeline/HotReloadManagerTests.cs b/source/CapriKit.Tests/AssetPipeline/HotReloadManagerTests.cs index 9cf12be..3f38da0 100644 --- a/source/CapriKit.Tests/AssetPipeline/HotReloadManagerTests.cs +++ b/source/CapriKit.Tests/AssetPipeline/HotReloadManagerTests.cs @@ -79,19 +79,31 @@ public async Task Update_RebuildFails() +/// +/// Builds like a until is set. Encoding runs on the +/// thread pool while tests flip the switch and read the counters from the main thread, so all three cross +/// that boundary explicitly. +/// internal sealed class TranscoderThatCanFail : TextTranscoder { + private volatile bool shouldFail; + private int attempts; + private int failedAttempts; - public bool ShouldFail { get; set; } = false; + public bool ShouldFail { get => shouldFail; set => shouldFail = value; } - /// Written on a thread pool thread, only safe to read once the rebuild completed. - public int FailedAttempts { get; private set; } + /// Every build the manager asked for, whether it succeeded or not. + public int Attempts => Volatile.Read(ref attempts); + + public int FailedAttempts => Volatile.Read(ref failedAttempts); public override Task Encode(AssetId id, IReadOnlyVirtualFileSystem fileSystem, IBufferWriter writer) { + Interlocked.Increment(ref attempts); + if (ShouldFail) { - FailedAttempts++; + Interlocked.Increment(ref failedAttempts); throw new InvalidOperationException("Rebuilding this asset failed on purpose"); } return base.Encode(id, fileSystem, writer); From bc09db6fee41f453e980f68d15a11d7272561566 Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Tue, 25 Aug 2026 23:20:57 +0200 Subject: [PATCH 48/53] more failure scenarios --- source/CapriKit.AssetPipeline/Asset.cs | 11 ++ .../AssetBundleLoader.cs | 57 ++++-- source/CapriKit.AssetPipeline/AssetHandle.cs | 24 ++- .../AssetLoadException.cs | 13 ++ source/CapriKit.AssetPipeline/AssetManager.cs | 48 +++-- source/CapriKit.AssetPipeline/README.md | 4 +- .../AssetPipeline/AssetManagerTests.cs | 166 ++++++++++++++---- 7 files changed, 254 insertions(+), 69 deletions(-) create mode 100644 source/CapriKit.AssetPipeline/AssetLoadException.cs diff --git a/source/CapriKit.AssetPipeline/Asset.cs b/source/CapriKit.AssetPipeline/Asset.cs index 23624af..c93cb80 100644 --- a/source/CapriKit.AssetPipeline/Asset.cs +++ b/source/CapriKit.AssetPipeline/Asset.cs @@ -26,6 +26,17 @@ public override string ToString() internal record Asset(AssetId Id, TAsset Value, AssetBuildMetaData BuildMetaData) where TAsset : class; +/// +/// One finished load request handed back to the main thread: either a materializer that puts the asset in +/// the cache and takes a lease on it, or the failure that stopped the load. +/// +internal readonly record struct LoadResult(AssetId Id, Func? Materialize, AssetLoadException? Failure) +{ + public static LoadResult Success(AssetId id, Func materialize) => new(id, materialize, null); + + public static LoadResult Failed(AssetLoadException failure) => new(failure.Asset, null, failure); +} + /// /// Record of the exact transcoder, settings and files used to build the asset. /// diff --git a/source/CapriKit.AssetPipeline/AssetBundleLoader.cs b/source/CapriKit.AssetPipeline/AssetBundleLoader.cs index 2a8e498..5d70552 100644 --- a/source/CapriKit.AssetPipeline/AssetBundleLoader.cs +++ b/source/CapriKit.AssetPipeline/AssetBundleLoader.cs @@ -16,7 +16,7 @@ internal AssetBundleBuilder(AssetManager assetManager) } /// - /// Each asset that needs to loaded returns a handle that can then put used in the lambda + /// Each asset that needs to load returns a handle that can then be used in the lambda /// for to describe how /// the asset bundle should actually be built. /// @@ -42,7 +42,7 @@ public AssetBundleLoader Build(Func(factory, Handles); foreach (var handle in Handles) { - bundle.Add(handle.Id); + bundle.Add(handle); handle.Owner = bundle; } @@ -51,16 +51,22 @@ public AssetBundleLoader Build(Func -/// Track the progress of loading the actual bundle and can be used to retrieve the actual bundle when finished. +/// Tracks the progress of loading the actual bundle and can be used to retrieve the actual bundle when finished. /// public abstract class AssetBundleLoader { - private readonly HashSet AssetSet = []; + private readonly List HandleList = []; - internal void Add(AssetId id) => AssetSet.Add(id); - internal bool IsActive { get; set; } = true; + internal void Add(AssetHandle handle) => HandleList.Add(handle); + + /// + /// Every handle in this bundle, failed ones included, kept for as long as the bundle lives. Unloading + /// counts handles rather than distinct assets because the pool hands out one lease per resolved handle: + /// an asset this bundle asked for twice holds two leases, and one that failed to load holds none. + /// + internal IReadOnlyList Handles => HandleList; - public IReadOnlySet Assets => AssetSet; + internal bool IsActive { get; set; } = true; } /// @@ -76,7 +82,8 @@ public sealed class AssetBundleLoader(Func - /// The number of assets that have completed loading, updated every time is called. + /// The number of assets that have arrived, successfully or not, updated every time + /// is called. /// public int Loaded => Total - Pending.Count; @@ -89,8 +96,10 @@ public sealed class AssetBundleLoader(Func - /// Checks whether the bundle finished loading, and if so builds returns it. - /// Threading: primary thread only. + /// Checks whether the bundle finished loading, and if so builds and returns it. The result is + /// cached so calling IsReady multiple times after loading finished is OK. + /// Throws a when an asset in this bundle could not be built or + /// loaded. Throws an if multiple assets failed to load. /// public bool IsReady([NotNullWhen(true)] out TBundle? value) { @@ -99,15 +108,33 @@ public bool IsReady([NotNullWhen(true)] out TBundle? value) for (var i = Pending.Count - 1; i >= 0; i--) { var handle = Pending[i]; - if (handle.IsResolved) + if (!handle.IsCompleted) { continue; } + + // An asset that failed has arrived too, it just arrived as an error instead of as a value. + LastCompletedItem = handle.Id; + Pending[i] = Pending[^1]; + Pending.RemoveAt(Pending.Count - 1); + } + + if (Pending.Count > 0) { value = default; return false; } + + // Everything arrived, so every lease this bundle will ever hold is settled and it is safe to report + // a failure. Handles keep their own error, so this needs no bookkeeping of its own and reports in + // the order the assets were requested. isReady stays false, so every later call throws again. + List? failures = null; + foreach (var handle in Handles) + { + if (handle.Error is not null) { - LastCompletedItem = handle.Id; - Pending[i] = Pending[^1]; - Pending.RemoveAt(Pending.Count - 1); + (failures ??= []).Add(handle.Error); } } - if (Pending.Count > 0) { value = default; return false; } + if (failures != null) + { + if (failures.Count == 1) { throw failures[0]; } + throw new AggregateException(failures); + } result = value = factory(new AssetHandleResolver(this)); isReady = true; diff --git a/source/CapriKit.AssetPipeline/AssetHandle.cs b/source/CapriKit.AssetPipeline/AssetHandle.cs index 92bd422..7acf1ae 100644 --- a/source/CapriKit.AssetPipeline/AssetHandle.cs +++ b/source/CapriKit.AssetPipeline/AssetHandle.cs @@ -10,11 +10,21 @@ public abstract class AssetHandle(AssetId id) public AssetId Id { get; } = id; private object? value; + private AssetLoadException? error; private volatile bool isResolved; internal AssetBundleLoader? Owner { get; set; } - internal bool IsResolved => isResolved; + + /// True once the asset arrived, whether it loaded successfully or failed. + internal bool IsCompleted => isResolved; + + /// The failure that stopped this asset from loading, null while loading and after success. + internal AssetLoadException? Error => error; + + /// True once the asset arrived successfully, which is also when it holds a lease on the pool. + internal bool IsLoaded => IsCompleted && error is null; + internal object? Value => value; internal void Resolve(object asset) @@ -23,6 +33,18 @@ internal void Resolve(object asset) value = asset; isResolved = true; } + + /// + /// Hands this asset's failure to whoever is waiting for it. The volatile write to isResolved happens + /// last so a reader that sees the handle complete also sees the error, which is the same ordering + /// relies on for its value. + /// + internal void Fail(AssetLoadException exception) + { + Debug.Assert(isResolved == false); + error = exception; + isResolved = true; + } } /// diff --git a/source/CapriKit.AssetPipeline/AssetLoadException.cs b/source/CapriKit.AssetPipeline/AssetLoadException.cs new file mode 100644 index 0000000..d8b4d8a --- /dev/null +++ b/source/CapriKit.AssetPipeline/AssetLoadException.cs @@ -0,0 +1,13 @@ +namespace CapriKit.AssetPipeline; + +/// +/// Thrown from when building or loading an asset failed. +/// +public sealed class AssetLoadException(AssetId asset, Exception innerException) + : Exception($"Failed to build or load asset: {asset}", innerException) +{ + /// + /// The asset that could not be built or loaded. + /// + public AssetId Asset { get; } = asset; +} diff --git a/source/CapriKit.AssetPipeline/AssetManager.cs b/source/CapriKit.AssetPipeline/AssetManager.cs index 1720fd5..e1f437c 100644 --- a/source/CapriKit.AssetPipeline/AssetManager.cs +++ b/source/CapriKit.AssetPipeline/AssetManager.cs @@ -18,7 +18,7 @@ public sealed partial class AssetManager : IDisposable private readonly HotReloadManager HotReloadManager; private readonly ConcurrentDictionary Transcoders; - private readonly LightweightChannel<(AssetId Id, Func Materializer)> Incoming; + private readonly LightweightChannel Incoming; private readonly Lock RequestLock; private readonly Dictionary> Outstanding; @@ -69,6 +69,10 @@ public AssetBundleBuilder CreateBundle() internal AssetHandle Load(AssetId id, TSettings settings) where TAsset : class { + // Look the transcoder up here, and before the cache check, so that a missing or mismatched + // registration always throws on the calling thread as the programmer error it is. Doing it inside + // the request would turn it into a failed asset load, and only for assets that are not cached yet. + var transcoder = GetTranscoder(); var handle = new AssetHandle(id); // At this time an asset is either already loaded, already requested or requested for the first time. @@ -92,11 +96,11 @@ internal AssetHandle Load(AssetId id, TSettings setti { // Request the asset Outstanding[id] = [handle]; - Task.Run(() => RequestAsset(id, settings)).FireAndForget( + Task.Run(() => RequestAsset(id, settings, transcoder)).FireAndForget( ex => { LogFailed(Logger, id); - Incoming.Write(ex); + Incoming.Write(LoadResult.Failed(new AssetLoadException(id, ex.SourceException))); }); } } @@ -108,17 +112,15 @@ internal AssetHandle Load(AssetId id, TSettings setti /// Performs the actual loading or building and loading of the asset. /// Threading: The caller has to guarantee that this method does not run concurrently for the same asset-id. /// - private async Task RequestAsset(AssetId id, TSettings settings) + private async Task RequestAsset(AssetId id, TSettings settings, IAssetTranscoder transcoder) where TAsset : class { - var transcoder = GetTranscoder(); - // Check if the asset can be loaded from an up-to-date build var build = await AssetDecoder.TryDecodeBuildMetaData(id, transcoder, FileSystem); if (build != default && IsUpToDate(transcoder, settings, build, FileSystem)) { var upToDateAsset = await AssetDecoder.Decode(id, transcoder, FileSystem); - Incoming.Write((id, () => TrackAndTakeLease(upToDateAsset, transcoder))); + Incoming.Write(LoadResult.Success(id, () => TrackAndTakeLease(upToDateAsset, transcoder))); LogLoadedFromFile(Logger, id); } else // If not, try to rebuild and load the asset @@ -130,26 +132,32 @@ private async Task RequestAsset(AssetId id, TSettings setting await AssetEncoder.Encode(id, transcoder, settings, FileSystem); var freshAsset = await AssetDecoder.Decode(id, transcoder, FileSystem); - Incoming.Write((id, () => TrackAndTakeLease(freshAsset, transcoder))); + Incoming.Write(LoadResult.Success(id, () => TrackAndTakeLease(freshAsset, transcoder))); LogBuildAndLoaded(Logger, id); } } /// - /// Unloads all assets in the bundle. + /// Unloads all assets in the bundle. A bundle that failed to load can be unloaded too: the assets in it + /// that did load are returned, and the ones that failed never took anything that needs returning. /// Threading: Unload updates the internal state of the bundle using a lock so that it is safe /// to unload the same bundle from multiple threads. /// - public void Unload(AssetBundleLoader bundle) // TODO: this should unload an assetbundle, not a loader, but that is not a formal class/interface yet and I don't want it to become more complicated to define an asset bundle. What to do? + public void Unload(AssetBundleLoader bundle) { + // TODO: this should unload an assetbundle, not a loader, but that is not a formal class/interface yet and I don't want it to become more complicated to define an asset bundle. What to do? + // TODO: Claude says unloading a bundle that is still loading leaks (fails safe, but isn't right), but is that correct? try { RequestLock.Enter(); if (bundle.IsActive) { - foreach (var asset in bundle.Assets) + // Give back exactly what was taken. The pool counts one lease per resolved handle, so an + // asset this bundle asked for twice has to be returned twice, and one that failed to load + // or never arrived has nothing to return. + foreach (var handle in bundle.Handles) { - Cache.Return(asset); + if (handle.IsLoaded) { Cache.Return(handle.Id); } } } } @@ -163,6 +171,8 @@ public void Unload(AssetBundleLoader bundle) // TODO: this should unload an asse /// /// Materializes assets that have finished loading, removes unused items from the cache /// and hot-reloads changed assets. + /// A failed load does not throw here, it is handed to the handles that were waiting for it and + /// surfaces from instead. /// Threading: Should only be called from the primary thread. /// public void Update() @@ -173,14 +183,16 @@ public void Update() { while (Incoming.TryRead(out var result)) { - var (id, trackAndTakeLease) = result; - var handles = Outstanding[id]; - foreach (var handle in handles) + // Whether it loaded or failed the request is over, so it stops accepting handles. Doing that + // here, under the lock that Load uses, is what stops a handle joining a dead request: it + // either made this list, or it misses and its own Load starts a fresh request. + if (!Outstanding.Remove(result.Id, out var waiting)) { continue; } + + foreach (var handle in waiting) { - var asset = trackAndTakeLease(); - handle.Resolve(asset); + if (result.Failure is not null) { handle.Fail(result.Failure); } + else { handle.Resolve(result.Materialize!()); } } - Outstanding.Remove(id); } } diff --git a/source/CapriKit.AssetPipeline/README.md b/source/CapriKit.AssetPipeline/README.md index f66bd37..3a265c9 100644 --- a/source/CapriKit.AssetPipeline/README.md +++ b/source/CapriKit.AssetPipeline/README.md @@ -39,9 +39,9 @@ if (loader.isReady(out var bundle)) } ``` -7. If you no longer need the assets, or if the program exists, return the bundle so the assets can be disposed. Loading and unloading uses reference counting to ensure that an asset is only truly disposed of it nobody references it anymore. To ensure everyone correctly unloads their assets an exception will be thrown if the `AssetManager` (and underlying `AssetPool`) are disposed without first unloading all asset bundles. +7. If you no longer need the assets, or if the program exists, return the loader so the assets can be disposed. Loading and unloading uses reference counting to ensure that an asset is only truly disposed of it nobody references it anymore. To ensure everyone correctly unloads their assets an exception will be thrown if the `AssetManager` (and underlying `AssetPool`) are disposed without first unloading all assets. ``` -assetManager.Unload(bundle); +assetManager.Unload(loader); ``` ## Hot reloading diff --git a/source/CapriKit.Tests/AssetPipeline/AssetManagerTests.cs b/source/CapriKit.Tests/AssetPipeline/AssetManagerTests.cs index 0687496..f160bd2 100644 --- a/source/CapriKit.Tests/AssetPipeline/AssetManagerTests.cs +++ b/source/CapriKit.Tests/AssetPipeline/AssetManagerTests.cs @@ -81,16 +81,10 @@ await Assert.That(() => } /// - /// FAILS ON PURPOSE: it asserts the behaviour we want, which the AssetManager does not have yet. - /// - /// A load that fails wedges its asset id for the rest of the manager's life. RequestAsset reports the - /// failure through the incoming channel, but only its success path writes a materializer, so Update - /// never reaches the `Outstanding.Remove(id)` that clears the request. The dead handle list survives, - /// and because Load hands every later request for that id to the existing list instead of starting a - /// new one, the asset can never be loaded again, not even after the cause of the failure is gone. - /// - /// The fix is to clear the Outstanding entry when a request fails, next to the `Incoming.Write(ex)` - /// in AssetManager.Load, so that the next Load starts a fresh request. + /// A failed load used to wedge its asset id for the rest of the manager's life: the failure never + /// cleared the entry in Outstanding, and because Load hands every later request for that id to the + /// existing list instead of starting a new one, the asset could never be loaded again. Update now + /// forgets the failed request before it rethrows, so a later Load starts over. /// [Test] public async Task Load_RetriesAnAssetWhoseFirstLoadFailed() @@ -108,21 +102,27 @@ public async Task Load_RetriesAnAssetWhoseFirstLoadFailed() var id = new AssetId(AssetFile); - // Act: the first load fails while building and Update rethrows that failure on the main thread + // Act: the first load fails while building. Update stays quiet about it, the failure is handed to + // the bundle that was waiting and surfaces from IsReady. var failedBuilder = assetManager.CreateBundle(); var failedHandle = failedBuilder.Load(id); var failedLoader = failedBuilder.Build(resolver => new TestBundle(resolver.Get(failedHandle))); - Exception? failure = null; + AssetLoadException? failure = null; await Assert.That(() => { - try { assetManager.Update(); } - catch (Exception ex) { failure = ex; } + // Not guarded on purpose: Update throwing here fails this test, which is the point + assetManager.Update(); + + try { failedLoader.IsReady(out _); } + catch (AssetLoadException ex) { failure = ex; } return failure is not null; }) .Eventually(v => v.IsTrue(), TimeSpan.FromSeconds(5)); - await Assert.That(failure).IsTypeOf(); + // The exception names the asset that failed and keeps the transcoder's own exception as the cause + await Assert.That(failure!.Asset).IsEqualTo(id); + await Assert.That(failure!.InnerException).IsTypeOf(); await Assert.That(transcoder.Attempts).IsEqualTo(1); // Act: take away the reason the build failed and ask for the very same asset again @@ -132,8 +132,8 @@ await Assert.That(() => var retryHandle = retryBuilder.Load(id); var retryLoader = retryBuilder.Build(resolver => new TestBundle(resolver.Get(retryHandle))); - // Control: a second, untouched asset requested at the same moment. Waiting for it proves that a - // failure does not stop the manager as a whole, so a red test really is about this one asset id. + // Control: a second, untouched asset requested at the same moment, to show that a failure never + // stopped the manager as a whole and that the retry above is what actually changed. await fileSystem.WriteAllText(HealthyFile, TranscoderText); var healthyBuilder = assetManager.CreateBundle(); @@ -147,40 +147,140 @@ await Assert.That(() => }) .Eventually(v => v.IsTrue(), TimeSpan.FromSeconds(5)); - // Pump until the retry has built, and stop early once it has so that the green path stays quick. - // Today it never happens and we spend the whole window, which is what makes the failure below crisp - // instead of a five second timeout somewhere else. - for (var i = 0; i < 200 && transcoder.Attempts < 3; i++) + // Assert: the retry really rebuilt the asset, so the failure, the control and the retry each + // reached the transcoder exactly once + TestBundle? retried = null; + await Assert.That(() => { assetManager.Update(); - await Task.Delay(10); - } + return retryLoader.IsReady(out retried); + }) + .Eventually(v => v.IsTrue(), TimeSpan.FromSeconds(5)); - // Assert: the failure, the control and the retry each reached the transcoder exactly once. - // THIS IS THE ASSERTION THAT FAILS TODAY: the retry never starts, so it stops at 2. await Assert.That(transcoder.Attempts).IsEqualTo(3); + await Assert.That(retried!.Asset.Text).IsEqualTo(TranscoderText); + + // The first bundle keeps reporting its failure rather than quietly never completing + await Assert.That(() => failedLoader.IsReady(out _)).Throws(); + + // Every bundle can be unloaded, the failed one included: it returns nothing because its only handle + // never took a lease. Getting that wrong would return the lease that retryLoader holds on the same + // asset, so the clean dispose below is what proves the counting is right. + assetManager.Unload(failedLoader); + assetManager.Unload(retryLoader); + assetManager.Unload(healthyLoader); + await Assert.That(() => assetManager.Dispose()).ThrowsNothing(); + } + + /// + /// Forgetting to register a transcoder is a programmer error rather than a broken asset, so it has to + /// surface on the calling thread instead of arriving as a failed load a few frames later. Looking the + /// transcoder up before the cache is checked keeps that true whether or not the asset happens to be + /// cached already. + /// + [Test] + public async Task Load_ThrowsOnTheCallingThreadWhenNoTranscoderIsRegistered() + { + var fileSystem = new InMemoryFileSystem().ScopedTo("C:/Test"); + await fileSystem.WriteAllText(AssetFile, TranscoderText); + + using var assetManager = new AssetManager(NullLoggerFactory.Instance, fileSystem); + var builder = assetManager.CreateBundle(); + + await Assert.That(() => builder.Load(new AssetId(AssetFile))).Throws(); + } + + /// + /// Leases are counted per handle, not per distinct asset, so a bundle that asks for the same asset + /// twice holds two leases and has to give back two. Unloading used to work from the distinct asset ids + /// and gave back one, which leaked the asset for the rest of the program. + /// + [Test] + public async Task Unload_ReturnsOneLeasePerHandle() + { + var fileSystem = new InMemoryFileSystem().ScopedTo("C:/Test"); + await fileSystem.WriteAllText(AssetFile, TranscoderText); + + var assetManager = new AssetManager(NullLoggerFactory.Instance, fileSystem); + assetManager.RegisterTranscoder(new TextTranscoder()); + + var id = new AssetId(AssetFile); + + var builder = assetManager.CreateBundle(); + var first = builder.Load(id); + var second = builder.Load(id); + var loader = builder.Build(resolver => new TwiceBundle(resolver.Get(first), resolver.Get(second))); + TwiceBundle? bundle = null; await Assert.That(() => { assetManager.Update(); - return retryLoader.IsReady(out _); + return loader.IsReady(out bundle); }) .Eventually(v => v.IsTrue(), TimeSpan.FromSeconds(5)); - // The first bundle stays unresolved even after the fix. Its exception already surfaced out of - // Update and a handle carries no way to report one, so those handles are simply dead. - await Assert.That(failedLoader.IsReady(out _)).IsFalse(); + // Both handles resolved to the single cached instance, but each of them took its own lease + await Assert.That(bundle!.Second).IsSameReferenceAs(bundle.First); - // Only the two resolved bundles hold a lease, so a clean dispose also proves that the failed - // request never leaked one of its own. - assetManager.Unload(retryLoader); - assetManager.Unload(healthyLoader); + assetManager.Unload(loader); + await Assert.That(() => assetManager.Dispose()).ThrowsNothing(); + } + + /// + /// The counterpart of . A transcoder that throws + /// while loading is fatal, but the very same transcoder throwing during a hot-reload must not be: the + /// asset that is already live is still perfectly usable, and taking the game down over a development + /// feature would be worse than the stale contents. + /// + [Test] + public async Task Update_DoesNotThrowWhenAHotReloadFails() + { + var fileSystem = new InMemoryFileSystem().ScopedTo("C:/Test"); + await fileSystem.WriteAllText(AssetFile, TranscoderText); + + var transcoder = new TranscoderThatCanFail(); + var assetManager = new AssetManager(NullLoggerFactory.Instance, fileSystem); + assetManager.RegisterTranscoder(transcoder); + + var builder = assetManager.CreateBundle(); + var handle = builder.Load(new AssetId(AssetFile)); + var loader = builder.Build(resolver => new TestBundle(resolver.Get(handle))); + + TestBundle? bundle = null; + await Assert.That(() => + { + assetManager.Update(); + return loader.IsReady(out bundle); + }) + .Eventually(v => v.IsTrue(), TimeSpan.FromSeconds(5)); + + // Act: make every rebuild fail, then change the file the asset was built from + transcoder.ShouldFail = true; + await fileSystem.WriteAllText(AssetFile, "Goodbye World"); + + // Pump past the hot-reload debounce until the rebuild has actually been attempted and failed + Exception? thrown = null; + await Assert.That(() => + { + try { assetManager.Update(); } + catch (Exception ex) { thrown = ex; } + return transcoder.FailedAttempts; + }) + .Eventually(v => v.IsGreaterThan(0), TimeSpan.FromSeconds(10)); + + // Assert: the failure stayed inside the hot-reload path and the asset kept its old contents + await Assert.That(thrown).IsNull(); + await Assert.That(bundle!.Asset.Text).IsEqualTo(TranscoderText); + + assetManager.Unload(loader); assetManager.Dispose(); } } internal record TestBundle(TextAsset Asset); +internal record TwiceBundle(TextAsset First, TextAsset Second); + internal sealed class TextAsset(string text) { public string Text { get; set; } = text; From d23041283e7b1eb541c1ad0a59f0802f222393aa Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Wed, 26 Aug 2026 20:24:15 +0200 Subject: [PATCH 49/53] better unloads --- .../{AssetBundleLoader.cs => AssetBundle.cs} | 360 ++++++++++-------- source/CapriKit.AssetPipeline/AssetHandle.cs | 4 +- source/CapriKit.AssetPipeline/AssetManager.cs | 134 ++++++- source/CapriKit.AssetPipeline/README.md | 16 +- .../AssetPipeline/AssetManagerTests.cs | 111 +++++- .../TestUtilities/CapturingLoggerFactory.cs | 35 ++ 6 files changed, 477 insertions(+), 183 deletions(-) rename source/CapriKit.AssetPipeline/{AssetBundleLoader.cs => AssetBundle.cs} (59%) create mode 100644 source/CapriKit.Tests/TestUtilities/CapturingLoggerFactory.cs diff --git a/source/CapriKit.AssetPipeline/AssetBundleLoader.cs b/source/CapriKit.AssetPipeline/AssetBundle.cs similarity index 59% rename from source/CapriKit.AssetPipeline/AssetBundleLoader.cs rename to source/CapriKit.AssetPipeline/AssetBundle.cs index 5d70552..5279ca9 100644 --- a/source/CapriKit.AssetPipeline/AssetBundleLoader.cs +++ b/source/CapriKit.AssetPipeline/AssetBundle.cs @@ -1,157 +1,205 @@ -using System.Diagnostics.CodeAnalysis; - -namespace CapriKit.AssetPipeline; - -/// -/// Use the builder to describe how an asset bundle should be built. -/// -public sealed class AssetBundleBuilder -{ - private readonly List Handles = []; - private readonly AssetManager assetManager; - - internal AssetBundleBuilder(AssetManager assetManager) - { - this.assetManager = assetManager; - } - - /// - /// Each asset that needs to load returns a handle that can then be used in the lambda - /// for to describe how - /// the asset bundle should actually be built. - /// - public AssetHandle Load(AssetId id, TSettings settings) - where TAsset : class - { - var handle = assetManager.Load(id, settings); - Handles.Add(handle); - return handle; - } - - /// - public AssetHandle Load(AssetId id) - where TAsset : class - => Load(id, default); - - /// - /// Create a loader that is used to check the loading progress of the bundle. - /// - public AssetBundleLoader Build(Func factory) - where TBundle : notnull - { - var bundle = new AssetBundleLoader(factory, Handles); - foreach (var handle in Handles) - { - bundle.Add(handle); - handle.Owner = bundle; - } - - return bundle; - } -} - -/// -/// Tracks the progress of loading the actual bundle and can be used to retrieve the actual bundle when finished. -/// -public abstract class AssetBundleLoader -{ - private readonly List HandleList = []; - - internal void Add(AssetHandle handle) => HandleList.Add(handle); - - /// - /// Every handle in this bundle, failed ones included, kept for as long as the bundle lives. Unloading - /// counts handles rather than distinct assets because the pool hands out one lease per resolved handle: - /// an asset this bundle asked for twice holds two leases, and one that failed to load holds none. - /// - internal IReadOnlyList Handles => HandleList; - - internal bool IsActive { get; set; } = true; -} - -/// -public sealed class AssetBundleLoader(Func factory, IReadOnlyList handles) - : AssetBundleLoader - where TBundle : notnull -{ - private readonly List Pending = [.. handles]; - - /// - /// The number of assets in this bundle. - /// - public int Total { get; } = handles.Count; - - /// - /// The number of assets that have arrived, successfully or not, updated every time - /// is called. - /// - public int Loaded => Total - Pending.Count; - - /// - /// The latest item that completed loading, updated every time is called. - /// - public AssetId? LastCompletedItem { get; private set; } - - private TBundle? result; - private bool isReady; - - /// - /// Checks whether the bundle finished loading, and if so builds and returns it. The result is - /// cached so calling IsReady multiple times after loading finished is OK. - /// Throws a when an asset in this bundle could not be built or - /// loaded. Throws an if multiple assets failed to load. - /// - public bool IsReady([NotNullWhen(true)] out TBundle? value) - { - if (isReady) { value = result!; return true; } - - for (var i = Pending.Count - 1; i >= 0; i--) - { - var handle = Pending[i]; - if (!handle.IsCompleted) { continue; } - - // An asset that failed has arrived too, it just arrived as an error instead of as a value. - LastCompletedItem = handle.Id; - Pending[i] = Pending[^1]; - Pending.RemoveAt(Pending.Count - 1); - } - - if (Pending.Count > 0) { value = default; return false; } - - // Everything arrived, so every lease this bundle will ever hold is settled and it is safe to report - // a failure. Handles keep their own error, so this needs no bookkeeping of its own and reports in - // the order the assets were requested. isReady stays false, so every later call throws again. - List? failures = null; - foreach (var handle in Handles) - { - if (handle.Error is not null) - { - (failures ??= []).Add(handle.Error); - } - } - - if (failures != null) - { - if (failures.Count == 1) { throw failures[0]; } - throw new AggregateException(failures); - } - - result = value = factory(new AssetHandleResolver(this)); - isReady = true; - return true; - } - - /// - /// Busy waits until the bundle finishes loading, then builds and returns it. - /// Threading: primary thread only. - /// - public TBundle WaitUntilReady() - { - var wait = new SpinWait(); - while (true) - { - if (IsReady(out var bundle)) { return bundle; } - wait.SpinOnce(); - } - } +using System.Diagnostics.CodeAnalysis; + +namespace CapriKit.AssetPipeline; + +/// +/// Use the builder to describe how an asset bundle should be built. +/// +public sealed class AssetBundleBuilder +{ + private readonly List Handles = []; + private readonly AssetManager assetManager; + + internal AssetBundleBuilder(AssetManager assetManager, string origin) + { + this.assetManager = assetManager; + Origin = origin; + assetManager.RegisterBuilder(this); + } + + /// + /// The file and line that created this builder, used to name it if its assets are never unloaded. + /// + internal string Origin { get; } + + /// + /// The number of assets requested so far. + /// + internal int RequestedAssets => Handles.Count; + + /// + /// Each asset that needs to load returns a handle that can then be used in the lambda + /// for to describe how + /// the asset bundle should actually be built. + /// + public AssetHandle Load(AssetId id, TSettings settings) + where TAsset : class + { + var handle = assetManager.Load(id, settings); + Handles.Add(handle); + return handle; + } + + /// + public AssetHandle Load(AssetId id) + where TAsset : class + => Load(id, default); + + /// + /// Creates the bundle that owns the requested assets and that is used to check their loading progress. + /// Always build a builder that loaded something: the assets of an abandoned builder belong to no bundle + /// and can therefore never be unloaded. + /// + public AssetBundle Build(Func factory) + where TBundle : notnull + { + var bundle = new AssetBundle(assetManager, Origin, factory, Handles); + foreach (var handle in Handles) + { + handle.Owner = bundle; + } + + assetManager.RegisterBundle(this, bundle); + return bundle; + } +} + +/// +/// A set of assets that were requested together and that are unloaded together. The bundle holds one lease +/// per handle for as long as it lives, disposing it hands every one of them back. The strongly typed value +/// a bundle produces is only its contents, this object owns the lifetime of the assets in it. +/// +public abstract class AssetBundle : IDisposable +{ + private readonly List HandleList; + private readonly AssetManager Manager; + + private protected AssetBundle(AssetManager manager, string origin, IReadOnlyList handles) + { + Manager = manager; + Origin = origin; + + // Copied on purpose: the builder that handed us this list stays usable and may load more assets + // into it, those belong to whatever bundle is built next and not to this one. + HandleList = [.. handles]; + } + + /// + /// Every handle in this bundle, failed ones included, kept for as long as the bundle lives. Unloading + /// counts handles rather than distinct assets because the pool hands out one lease per resolved handle: + /// an asset this bundle asked for twice holds two leases, and one that failed to load holds none. + /// + internal IReadOnlyList Handles => HandleList; + + internal bool IsActive { get; set; } = true; + + /// + /// The file and line that created this bundle, used to name it if it is never unloaded. + /// + internal string Origin { get; } + + /// + /// Unloads this bundle, see . + /// Unloading a bundle twice is safe, the second call does nothing. + /// + public void Dispose() => Manager.Unload(this); +} + +/// +public sealed class AssetBundle : AssetBundle + where TBundle : notnull +{ + private readonly Func Factory; + private readonly List Pending; + + private TBundle? result; + private bool isReady; + + internal AssetBundle(AssetManager manager, string origin, Func factory, IReadOnlyList handles) + : base(manager, origin, handles) + { + Factory = factory; + Pending = [.. handles]; + Total = handles.Count; + } + + /// + /// The number of assets in this bundle. + /// + public int Total { get; } + + /// + /// The number of assets that have arrived, successfully or not, updated every time + /// is called. + /// + public int Loaded => Total - Pending.Count; + + /// + /// The latest item that completed loading, updated every time is called. + /// + public AssetId? LastCompletedItem { get; private set; } + + /// + /// Checks whether the bundle finished loading, and if so builds and returns it. The result is + /// cached so calling IsReady multiple times after loading finished is OK. + /// Throws a when an asset in this bundle could not be built or + /// loaded. Throws an if multiple assets failed to load. + /// Throws an once the bundle has been unloaded. + /// + public bool IsReady([NotNullWhen(true)] out TBundle? value) + { + // An unloaded bundle gave its leases back, so the assets it would hand out may already be disposed. + ObjectDisposedException.ThrowIf(!IsActive, this); + + if (isReady) { value = result!; return true; } + + for (var i = Pending.Count - 1; i >= 0; i--) + { + var handle = Pending[i]; + if (!handle.IsCompleted) { continue; } + + // An asset that failed has arrived too, it just arrived as an error instead of as a value. + LastCompletedItem = handle.Id; + Pending[i] = Pending[^1]; + Pending.RemoveAt(Pending.Count - 1); + } + + if (Pending.Count > 0) { value = default; return false; } + + // Everything arrived, so every lease this bundle will ever hold is settled and it is safe to report + // a failure. Handles keep their own error, so this needs no bookkeeping of its own and reports in + // the order the assets were requested. isReady stays false, so every later call throws again. + List? failures = null; + foreach (var handle in Handles) + { + if (handle.Error is not null) + { + (failures ??= []).Add(handle.Error); + } + } + + if (failures != null) + { + if (failures.Count == 1) { throw failures[0]; } + throw new AggregateException(failures); + } + + result = value = Factory(new AssetHandleResolver(this)); + isReady = true; + return true; + } + + /// + /// Busy waits until the bundle finishes loading, then builds and returns it. + /// Threading: primary thread only. + /// + public TBundle WaitUntilReady() + { + var wait = new SpinWait(); + while (true) + { + if (IsReady(out var bundle)) { return bundle; } + wait.SpinOnce(); + } + } } diff --git a/source/CapriKit.AssetPipeline/AssetHandle.cs b/source/CapriKit.AssetPipeline/AssetHandle.cs index 7acf1ae..d1b640c 100644 --- a/source/CapriKit.AssetPipeline/AssetHandle.cs +++ b/source/CapriKit.AssetPipeline/AssetHandle.cs @@ -14,7 +14,7 @@ public abstract class AssetHandle(AssetId id) private volatile bool isResolved; - internal AssetBundleLoader? Owner { get; set; } + internal AssetBundle? Owner { get; set; } /// True once the asset arrived, whether it loaded successfully or failed. internal bool IsCompleted => isResolved; @@ -53,7 +53,7 @@ public sealed class AssetHandle(AssetId id) : AssetHandle(id) { } /// /// Helper class for resolving loaded assets from their asset handle. /// -public sealed class AssetHandleResolver(AssetBundleLoader owner) +public sealed class AssetHandleResolver(AssetBundle owner) { public TValue Get(AssetHandle promise) { diff --git a/source/CapriKit.AssetPipeline/AssetManager.cs b/source/CapriKit.AssetPipeline/AssetManager.cs index e1f437c..29bec19 100644 --- a/source/CapriKit.AssetPipeline/AssetManager.cs +++ b/source/CapriKit.AssetPipeline/AssetManager.cs @@ -4,11 +4,12 @@ using Microsoft.Extensions.Logging; using System.Buffers; using System.Collections.Concurrent; +using System.Runtime.CompilerServices; namespace CapriKit.AssetPipeline; /// -/// Manages the building, loading, chaching, clean-up and hot-reloading of assets. +/// Manages the building, loading, caching, clean-up and hot-reloading of assets. /// public sealed partial class AssetManager : IDisposable { @@ -22,6 +23,11 @@ public sealed partial class AssetManager : IDisposable private readonly Lock RequestLock; private readonly Dictionary> Outstanding; + // Everything that holds, or is going to hold, leases. Kept so that Dispose can name whoever forgot + // to unload instead of only reporting that some number of assets was left behind. + private readonly HashSet LiveBundles; + private readonly HashSet UnbuiltBundles; + public AssetManager(ILoggerFactory logger, ScopedFileSystem fileSystem) { Logger = logger.CreateLogger(); @@ -32,6 +38,8 @@ public AssetManager(ILoggerFactory logger, ScopedFileSystem fileSystem) Incoming = new(); RequestLock = new(); Outstanding = []; + LiveBundles = []; + UnbuiltBundles = []; } /// @@ -51,27 +59,48 @@ public void RegisterTranscoder(IAssetTranscoder - /// Use to defines a bundle of assets to load. + /// Use to defines a bundle of assets to load. The call site is captured so that a bundle that is never + /// unloaded can point at the code that created it, callers should not pass the two arguments themselves. + /// Threading: thread-safe. + /// + public AssetBundleBuilder CreateBundle([CallerFilePath] string file = "", [CallerLineNumber] int line = 0) + { + return new AssetBundleBuilder(this, $"{Path.GetFileName(file.AsSpan())}:{line}"); + } + + /// + /// Registers a builder as a future owner of leases, see . + /// Threading: thread-safe. + /// + internal void RegisterBuilder(AssetBundleBuilder builder) + { + lock (RequestLock) { UnbuiltBundles.Add(builder); } + } + + /// + /// Hands ownership of the leases from the builder to the bundle it built. From here on the bundle is + /// what has to be unloaded, and what reports if that never happens. /// Threading: thread-safe. /// - public AssetBundleBuilder CreateBundle() + internal void RegisterBundle(AssetBundleBuilder builder, AssetBundle bundle) { - return new AssetBundleBuilder(this); + lock (RequestLock) + { + UnbuiltBundles.Remove(builder); + LiveBundles.Add(bundle); + } } /// /// Starts loading an asset. The asset will either be loaded from the cache, from disk, or rebuild and then loaded. - /// The caller gets a handle to be used in an which can be resolved - /// to the actual asset when loading finishes using + /// The caller gets a handle to be used in an which can be resolved + /// to the actual asset when loading finishes using /// Threading: thread-safe, can be called from any thread concurrently. This method guarantees that the same asset /// is not loaded multiple times concurrently. /// internal AssetHandle Load(AssetId id, TSettings settings) where TAsset : class { - // Look the transcoder up here, and before the cache check, so that a missing or mismatched - // registration always throws on the calling thread as the programmer error it is. Doing it inside - // the request would turn it into a failed asset load, and only for assets that are not cached yet. var transcoder = GetTranscoder(); var handle = new AssetHandle(id); @@ -139,14 +168,14 @@ private async Task RequestAsset(AssetId id, TSettings setting /// /// Unloads all assets in the bundle. A bundle that failed to load can be unloaded too: the assets in it - /// that did load are returned, and the ones that failed never took anything that needs returning. + /// that did load are returned, and the ones that failed never took anything that needs returning. So can + /// a bundle that is still loading, the leases its assets take when they do arrive are returned right away. + /// Unloading the same bundle twice is safe, the second call does nothing. /// Threading: Unload updates the internal state of the bundle using a lock so that it is safe /// to unload the same bundle from multiple threads. /// - public void Unload(AssetBundleLoader bundle) + public void Unload(AssetBundle bundle) { - // TODO: this should unload an assetbundle, not a loader, but that is not a formal class/interface yet and I don't want it to become more complicated to define an asset bundle. What to do? - // TODO: Claude says unloading a bundle that is still loading leaks (fails safe, but isn't right), but is that correct? try { RequestLock.Enter(); @@ -159,10 +188,13 @@ public void Unload(AssetBundleLoader bundle) { if (handle.IsLoaded) { Cache.Return(handle.Id); } } + + LiveBundles.Remove(bundle); } } finally { + // Marks the handles that have not arrived yet as unwanted, Update returns their leases for us. bundle.IsActive = false; RequestLock.Exit(); } @@ -172,7 +204,7 @@ public void Unload(AssetBundleLoader bundle) /// Materializes assets that have finished loading, removes unused items from the cache /// and hot-reloads changed assets. /// A failed load does not throw here, it is handed to the handles that were waiting for it and - /// surfaces from instead. + /// surfaces from instead. /// Threading: Should only be called from the primary thread. /// public void Update() @@ -191,7 +223,16 @@ public void Update() foreach (var handle in waiting) { if (result.Failure is not null) { handle.Fail(result.Failure); } - else { handle.Resolve(result.Materialize!()); } + else + { + handle.Resolve(result.Materialize!()); + + // The bundle was unloaded while this asset was still on its way. Unload could not + // return the lease that resolving just took, because back then there was nothing to + // return yet, so give it back here instead. Materializing first and returning after + // is deliberate: it routes the asset through the pool, which is what disposes it. + if (handle.Owner is { IsActive: false }) { Cache.Return(handle.Id); } + } } } } @@ -200,14 +241,71 @@ public void Update() HotReloadManager.Update(); } + /// + /// Shuts the asset manager down. Bundles that were never unloaded are reported but deliberately not + /// unloaded for you: this only runs at shutdown, where there is nobody left to hand the assets to, so + /// quietly cleaning up after the caller would only hide the bug that the log and the exception report. + /// Threading: Should only be called from the primary thread. + /// public void Dispose() { // Dispose the hot-reload manager first so that it can release any reference to // assets it might still hold. HotReloadManager.Dispose(); + + DrainIncoming(); + ReportAssetsThatWereNeverUnloaded(); + Cache.Dispose(); } + /// + /// Settles the loads that finished after the last . Those assets were decoded but + /// never reached a handle, so nobody leases them and nothing would ever dispose them. Taking the lease + /// and immediately giving it back moves them through the pool, which does dispose them. + /// + private void DrainIncoming() + { + lock (RequestLock) + { + while (Incoming.TryRead(out var result)) + { + Outstanding.Remove(result.Id); + + // A failed load never built anything, so there is nothing to dispose of. + if (result.Failure is not null) { continue; } + + result.Materialize!(); + Cache.Return(result.Id); + } + } + } + + /// + /// Names every bundle that still holds leases, and every builder whose assets never got a bundle to + /// belong to, so that the leak the throws about can be traced back to its code. + /// + private void ReportAssetsThatWereNeverUnloaded() + { + lock (RequestLock) + { + foreach (var bundle in LiveBundles) + { + LogBundleNotUnloaded(Logger, bundle.Origin, bundle.Handles.Count); + } + + foreach (var builder in UnbuiltBundles) + { + // A builder that never loaded anything is simply unused, only one that did is a problem: + // its handles have no bundle to be unloaded through. + if (builder.RequestedAssets > 0) + { + LogBundleNeverBuilt(Logger, builder.Origin, builder.RequestedAssets); + } + } + } + } + // Thread safe, only touches the file system and uses thread safe transcoder methods and properties. private static bool IsUpToDate(IAssetTranscoder transcoder, TSettings settings, AssetBuildMetaData build, IReadOnlyVirtualFileSystem fileSystem) where TAsset : class @@ -291,4 +389,10 @@ private IAssetTranscoder GetTranscoder() whe [LoggerMessage(Level = LogLevel.Error, Message = "Building or loading asset: {asset} failed.")] private static partial void LogFailed(ILogger logger, AssetId asset); + + [LoggerMessage(Level = LogLevel.Error, Message = "The asset bundle created at {origin} was never unloaded, it still holds {assets} asset(s)")] + private static partial void LogBundleNotUnloaded(ILogger logger, string origin, int assets); + + [LoggerMessage(Level = LogLevel.Error, Message = "The asset bundle builder created at {origin} loaded {assets} asset(s) but was never built, so those assets could never be unloaded")] + private static partial void LogBundleNeverBuilt(ILogger logger, string origin, int assets); } diff --git a/source/CapriKit.AssetPipeline/README.md b/source/CapriKit.AssetPipeline/README.md index 3a265c9..8e3da15 100644 --- a/source/CapriKit.AssetPipeline/README.md +++ b/source/CapriKit.AssetPipeline/README.md @@ -10,7 +10,7 @@ var assetManager = new AssetManager(LoggerFactory, ScopedFileSystem); assetManager.RegisterTranscoder(new VertexShaderTranscoder(GraphicsDevice)); ``` -2. Define a bundle that reflects the strongly typed assets that you want to load. +2. Define the contents of a bundle: a plain type that reflects the strongly typed assets that you want to load. It needs no base class or interface, the `AssetBundle` that the pipeline hands you owns the lifetime of the assets in it. ``` record UIBundle(PixelShader Shader, Texture2D Font); ``` @@ -21,9 +21,9 @@ var builder = assetManager.CreateBundle(); var shaderHandle = builder.Load(new AssetId("./shader.hlsl")); var fontHandle = builder.Load(new AssetId("./"robo.png")); ``` -4. Use the obtained handles to describe how the bundle can be construct once building and loading finishes. +4. Use the obtained handles to describe how the bundle can be constructed once building and loading finishes. ``` -var loader = builder.Build(r => new UIBundle(r.get(shaderHandle), r.get(fontHandle)); +using var bundle = builder.Build(r => new UIBundle(r.get(shaderHandle), r.get(fontHandle)); ``` 5. Do not forget to update the asset manager each frame so that it can manage the loading tasks and other bookkeeping. @@ -31,19 +31,21 @@ var loader = builder.Build(r => new UIBundle(r.get(shaderHandle), r.get(fontHand assetManager.Update(); ``` -6. Check each frame if loading finished and obtain your strongly typed bundle +6. Check each frame if loading finished and obtain your strongly typed contents ``` -if (loader.isReady(out var bundle)) +if (bundle.isReady(out var ui)) { // Yay! } ``` -7. If you no longer need the assets, or if the program exists, return the loader so the assets can be disposed. Loading and unloading uses reference counting to ensure that an asset is only truly disposed of it nobody references it anymore. To ensure everyone correctly unloads their assets an exception will be thrown if the `AssetManager` (and underlying `AssetPool`) are disposed without first unloading all assets. +7. If you no longer need the assets, or if the program exits, dispose the bundle so the assets can be disposed. Loading and unloading uses reference counting to ensure that an asset is only truly disposed of it nobody references it anymore. Unloading a bundle that is still loading is fine, the assets that are still on their way are returned as soon as they arrive. ``` -assetManager.Unload(loader); +bundle.Dispose(); // or, the same thing: assetManager.Unload(bundle); ``` +To ensure everyone correctly unloads their assets an exception will be thrown if the `AssetManager` (and underlying `AssetPool`) are disposed without first unloading all assets. The assets that were left behind are deliberately not unloaded for you, that would only hide the bug. To help you find it the asset manager logs an error naming every bundle that was never unloaded, and the file and line that created it. + ## Hot reloading Hot reloading happens automatically if the files that were used to build the asset are present and change. diff --git a/source/CapriKit.Tests/AssetPipeline/AssetManagerTests.cs b/source/CapriKit.Tests/AssetPipeline/AssetManagerTests.cs index f160bd2..1c36da3 100644 --- a/source/CapriKit.Tests/AssetPipeline/AssetManagerTests.cs +++ b/source/CapriKit.Tests/AssetPipeline/AssetManagerTests.cs @@ -4,6 +4,7 @@ using CapriKit.Tests.TestUtilities; using Microsoft.Extensions.Logging.Abstractions; using System.Buffers; +using System.Collections.Concurrent; namespace CapriKit.Tests.AssetPipeline; @@ -62,7 +63,7 @@ await Assert.That(() => await Assert.That(bundle.Asset.Text).IsEqualTo(TranscoderText); // Load again to verify loading the same thing twice gives us the cached value - var altBuilder = new AssetBundleBuilder(assetManager); + var altBuilder = assetManager.CreateBundle(); var altHandle = altBuilder.Load(id, default); var altLoader = altBuilder.Build(resolver => new TestBundle(resolver.Get(altHandle))); @@ -77,7 +78,89 @@ await Assert.That(() => await Assert.That(altBundle).IsNotNull(); await Assert.That(altBundle.Asset).IsSameReferenceAs(bundle.Asset); - // TODO: test the dispose path. Check the errors if assets are not returned and that returning both bundles correctly disposes everything. + // Both bundles lease the one shared instance, so it only really goes away once both gave it back + assetManager.Unload(loader); + assetManager.Update(); + await Assert.That(bundle.Asset.IsDisposed).IsFalse(); + + assetManager.Unload(altLoader); + assetManager.Update(); + await Assert.That(bundle.Asset.IsDisposed).IsTrue(); + + // Nothing is left over, so shutting down is quiet + await Assert.That(() => assetManager.Dispose()).ThrowsNothing(); + } + + /// + /// Unloading a bundle whose assets are still on their way used to leak them: the handles had taken no + /// lease yet so Unload had nothing to return, but the load still landed in a later Update and took one + /// that nobody would ever give back. Update now returns the lease of a handle whose bundle is gone. + /// + [Test] + public async Task Unload_ReturnsTheLeaseOfAnAssetThatWasStillLoading() + { + var fileSystem = new InMemoryFileSystem().ScopedTo("C:/Test"); + await fileSystem.WriteAllText(AssetFile, TranscoderText); + + var transcoder = new TrackingTextTranscoder(); + var assetManager = new AssetManager(NullLoggerFactory.Instance, fileSystem); + assetManager.RegisterTranscoder(transcoder); + + var builder = assetManager.CreateBundle(); + var handle = builder.Load(new AssetId(AssetFile)); + var bundle = builder.Build(resolver => new TestBundle(resolver.Get(handle))); + + // Act: unload before the first Update, so the asset is still loading and holds no lease yet. + // Disposing the bundle is the same thing as unloading it, and is how this is meant to be written. + bundle.Dispose(); + + // The load still finishes and still takes its lease, the manager has to hand that one straight back + await Assert.That(() => + { + assetManager.Update(); + return transcoder.Decoded.Count == 1 && transcoder.Decoded.All(asset => asset.IsDisposed); + }) + .Eventually(v => v.IsTrue(), TimeSpan.FromSeconds(5)); + + // The bundle no longer owns its assets, so it refuses to hand them out rather than serving disposed ones + await Assert.That(() => bundle.IsReady(out _)).Throws(); + + await Assert.That(() => assetManager.Dispose()).ThrowsNothing(); + } + + /// + /// Everything that is loaded has to be unloaded before the game quits. The pool notices when that did + /// not happen but can only count the assets left behind, so the manager names the bundle they belong to + /// and the line that created it. It deliberately does not unload them: at shutdown that would only hide + /// the bug from whoever has to fix it. + /// + [Test] + public async Task Dispose_ReportsBundlesThatWereNeverUnloaded() + { + var fileSystem = new InMemoryFileSystem().ScopedTo("C:/Test"); + await fileSystem.WriteAllText(AssetFile, TranscoderText); + + var logger = new CapturingLoggerFactory(); + var assetManager = new AssetManager(logger, fileSystem); + assetManager.RegisterTranscoder(new TrackingTextTranscoder()); + + var builder = assetManager.CreateBundle(); + var handle = builder.Load(new AssetId(AssetFile)); + var bundle = builder.Build(resolver => new TestBundle(resolver.Get(handle))); + + await Assert.That(() => + { + assetManager.Update(); + return bundle.IsReady(out _); + }) + .Eventually(v => v.IsTrue(), TimeSpan.FromSeconds(5)); + + // Act: quit without unloading anything + await Assert.That(() => assetManager.Dispose()).Throws(); + + var report = logger.Messages.SingleOrDefault(message => message.Contains("was never unloaded")); + await Assert.That(report).IsNotNull(); + await Assert.That(report!.Contains($"{nameof(AssetManagerTests)}.cs:")).IsTrue(); } /// @@ -281,9 +364,31 @@ internal record TestBundle(TextAsset Asset); internal record TwiceBundle(TextAsset First, TextAsset Second); -internal sealed class TextAsset(string text) +internal sealed class TextAsset(string text) : IDisposable { public string Text { get; set; } = text; + + /// Lets tests see whether the pool really let go of this asset. + public bool IsDisposed { get; private set; } + + public void Dispose() => IsDisposed = true; +} + +/// +/// Hands out the assets it decoded so that a test can look at an asset the asset manager never gave it. +/// +internal sealed class TrackingTextTranscoder : TextTranscoder +{ + private readonly ConcurrentBag decoded = []; + + public IReadOnlyCollection Decoded => decoded; + + public override TextAsset Decode(AssetId id, ref SequenceReader reader) + { + var asset = base.Decode(id, ref reader); + decoded.Add(asset); + return asset; + } } internal class TextTranscoder() : NoSettingsTranscoder(Guid.Parse("{6E4A1D0C-1F73-4C4E-9D2E-0B7F5C6A9E31}"), 1) diff --git a/source/CapriKit.Tests/TestUtilities/CapturingLoggerFactory.cs b/source/CapriKit.Tests/TestUtilities/CapturingLoggerFactory.cs new file mode 100644 index 0000000..241fd1d --- /dev/null +++ b/source/CapriKit.Tests/TestUtilities/CapturingLoggerFactory.cs @@ -0,0 +1,35 @@ +using Microsoft.Extensions.Logging; + +namespace CapriKit.Tests.TestUtilities; + +/// +/// Logger factory that keeps every formatted message, for tests that assert on a diagnostic instead of on +/// a return value. Messages may be written from any thread, reading them is meant for the main thread. +/// +internal sealed class CapturingLoggerFactory : ILoggerFactory +{ + private readonly List messages = []; + + public IReadOnlyList Messages + { + get { lock (messages) { return [.. messages]; } } + } + + public ILogger CreateLogger(string categoryName) => new CapturingLogger(messages); + + public void AddProvider(ILoggerProvider provider) { } + + public void Dispose() { } + + private sealed class CapturingLogger(List messages) : ILogger + { + public IDisposable? BeginScope(TState state) where TState : notnull => null; + + public bool IsEnabled(LogLevel logLevel) => true; + + public void Log(LogLevel logLevel, EventId eventId, TState state, Exception? exception, Func formatter) + { + lock (messages) { messages.Add(formatter(state, exception)); } + } + } +} From ce0ee4d21f087aa993b16f20344371aa73e50362 Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Wed, 26 Aug 2026 22:04:30 +0200 Subject: [PATCH 50/53] Yay --- source/CapriKit.AssetPipeline/AssetBundle.cs | 220 +++++++++--------- source/CapriKit.AssetPipeline/AssetManager.cs | 62 ++--- source/CapriKit.AssetPipeline/README.md | 14 +- .../AssetPipeline/AssetManagerTests.cs | 115 ++++----- 4 files changed, 190 insertions(+), 221 deletions(-) diff --git a/source/CapriKit.AssetPipeline/AssetBundle.cs b/source/CapriKit.AssetPipeline/AssetBundle.cs index 5279ca9..d94ed12 100644 --- a/source/CapriKit.AssetPipeline/AssetBundle.cs +++ b/source/CapriKit.AssetPipeline/AssetBundle.cs @@ -3,40 +3,67 @@ namespace CapriKit.AssetPipeline; /// -/// Use the builder to describe how an asset bundle should be built. +/// A set of assets that are loaded together and unloaded together. The bundle holds one lease per asset it +/// loaded for as long as it lives, disposing it hands every one of them back. Use to +/// describe the strongly typed value those assets add up to, the bundle itself owns their lifetime. +/// Threading: create, load and build from one thread, usually the primary one. /// -public sealed class AssetBundleBuilder +public sealed class AssetBundle : IDisposable { - private readonly List Handles = []; - private readonly AssetManager assetManager; + private readonly List HandleList = []; + private readonly AssetManager Manager; - internal AssetBundleBuilder(AssetManager assetManager, string origin) + internal AssetBundle(AssetManager manager, string origin) { - this.assetManager = assetManager; + Manager = manager; Origin = origin; - assetManager.RegisterBuilder(this); } /// - /// The file and line that created this builder, used to name it if its assets are never unloaded. + /// The number of assets in this bundle. /// - internal string Origin { get; } + public int Total => HandleList.Count; /// - /// The number of assets requested so far. + /// The number of assets that have arrived, successfully or not, updated every time + /// is called. /// - internal int RequestedAssets => Handles.Count; + public int Loaded { get; private set; } /// - /// Each asset that needs to load returns a handle that can then be used in the lambda - /// for to describe how - /// the asset bundle should actually be built. + /// The last asset that had arrived the last time was + /// called, in the order the assets were requested. Meant to put a name on a loading screen, not to + /// report the exact order in which assets finished. + /// + public AssetId? LastCompletedItem { get; private set; } + + /// + /// Every handle in this bundle, failed ones included, kept for as long as the bundle lives. Unloading + /// counts handles rather than distinct assets because the pool hands out one lease per resolved handle: + /// an asset this bundle asked for twice holds two leases, and one that failed to load holds none. + /// + internal IReadOnlyList Handles => HandleList; + + internal bool IsActive { get; set; } = true; + + /// + /// The file and line that created this bundle, used to name it if it is never unloaded. + /// + internal string Origin { get; } + + /// + /// Starts building (if needed) and loading an asset, and adds it to this bundle. The handle it returns + /// is used in the lambda for to + /// describe how the assets in this bundle add up to one strongly typed value. Load everything the + /// bundle needs before building it, an asset loaded afterwards still belongs to the bundle but the + /// loaders that were already built cannot see it. /// public AssetHandle Load(AssetId id, TSettings settings) where TAsset : class { - var handle = assetManager.Load(id, settings); - Handles.Add(handle); + var handle = Manager.Load(id, settings); + handle.Owner = this; + HandleList.Add(handle); return handle; } @@ -46,98 +73,94 @@ public AssetHandle Load(AssetId id) => Load(id, default); /// - /// Creates the bundle that owns the requested assets and that is used to check their loading progress. - /// Always build a builder that loaded something: the assets of an abandoned builder belong to no bundle - /// and can therefore never be unloaded. + /// Creates the loader that reports the progress of this bundle and that builds the strongly typed value + /// its assets add up to. The loader borrows the bundle, it does not own it: this bundle stays the thing + /// that has to be unloaded, which is why building twice is harmless. /// - public AssetBundle Build(Func factory) + public AssetBundleLoader Build(Func factory) where TBundle : notnull - { - var bundle = new AssetBundle(assetManager, Origin, factory, Handles); - foreach (var handle in Handles) - { - handle.Owner = bundle; - } - - assetManager.RegisterBundle(this, bundle); - return bundle; - } -} + => new(this, factory); -/// -/// A set of assets that were requested together and that are unloaded together. The bundle holds one lease -/// per handle for as long as it lives, disposing it hands every one of them back. The strongly typed value -/// a bundle produces is only its contents, this object owns the lifetime of the assets in it. -/// -public abstract class AssetBundle : IDisposable -{ - private readonly List HandleList; - private readonly AssetManager Manager; + /// + /// Unloads this bundle, see . + /// Unloading a bundle twice is safe, the second call does nothing. + /// + public void Dispose() => Manager.Unload(this); - private protected AssetBundle(AssetManager manager, string origin, IReadOnlyList handles) + /// + /// Updates and and reports whether every asset in + /// the bundle arrived, successfully or not. Walks all handles rather than keeping a list of the pending + /// ones: a bundle holds a handful of assets, and one list is easier to reason about than two that have + /// to agree with each other. + /// + internal bool AllAssetsArrived() { - Manager = manager; - Origin = origin; + var arrived = 0; + foreach (var handle in HandleList) + { + // An asset that failed has arrived too, it just arrived as an error instead of as a value. + if (handle.IsCompleted) + { + arrived++; + LastCompletedItem = handle.Id; + } + } - // Copied on purpose: the builder that handed us this list stays usable and may load more assets - // into it, those belong to whatever bundle is built next and not to this one. - HandleList = [.. handles]; + Loaded = arrived; + return arrived == HandleList.Count; } /// - /// Every handle in this bundle, failed ones included, kept for as long as the bundle lives. Unloading - /// counts handles rather than distinct assets because the pool hands out one lease per resolved handle: - /// an asset this bundle asked for twice holds two leases, and one that failed to load holds none. + /// Reports the assets in this bundle that could not be built or loaded, in the order they were + /// requested. Only call this once is true: until then an asset might + /// still take a lease, and reporting a failure before that would leave it unaccounted for. /// - internal IReadOnlyList Handles => HandleList; - - internal bool IsActive { get; set; } = true; + internal void ThrowOnFailedAssets() + { + List? failures = null; + foreach (var handle in HandleList) + { + if (handle.Error is not null) + { + (failures ??= []).Add(handle.Error); + } + } - /// - /// The file and line that created this bundle, used to name it if it is never unloaded. - /// - internal string Origin { get; } + if (failures is null) { return; } + if (failures.Count == 1) { throw failures[0]; } - /// - /// Unloads this bundle, see . - /// Unloading a bundle twice is safe, the second call does nothing. - /// - public void Dispose() => Manager.Unload(this); + throw new AggregateException(failures); + } } -/// -public sealed class AssetBundle : AssetBundle +/// +/// Tracks the loading progress of an and builds the strongly typed value its +/// assets add up to once they all arrived. Purely a view on the bundle: unloading goes through the bundle, +/// so a loader that is dropped costs nothing. +/// +public sealed class AssetBundleLoader where TBundle : notnull { + private readonly AssetBundle Bundle; private readonly Func Factory; - private readonly List Pending; private TBundle? result; private bool isReady; - internal AssetBundle(AssetManager manager, string origin, Func factory, IReadOnlyList handles) - : base(manager, origin, handles) + internal AssetBundleLoader(AssetBundle bundle, Func factory) { + Bundle = bundle; Factory = factory; - Pending = [.. handles]; - Total = handles.Count; } - /// - /// The number of assets in this bundle. - /// - public int Total { get; } + /// + public int Total => Bundle.Total; - /// - /// The number of assets that have arrived, successfully or not, updated every time - /// is called. - /// - public int Loaded => Total - Pending.Count; + /// + public int Loaded => Bundle.Loaded; - /// - /// The latest item that completed loading, updated every time is called. - /// - public AssetId? LastCompletedItem { get; private set; } + /// + public AssetId? LastCompletedItem => Bundle.LastCompletedItem; /// /// Checks whether the bundle finished loading, and if so builds and returns it. The result is @@ -149,42 +172,17 @@ internal AssetBundle(AssetManager manager, string origin, Func= 0; i--) - { - var handle = Pending[i]; - if (!handle.IsCompleted) { continue; } - - // An asset that failed has arrived too, it just arrived as an error instead of as a value. - LastCompletedItem = handle.Id; - Pending[i] = Pending[^1]; - Pending.RemoveAt(Pending.Count - 1); - } - - if (Pending.Count > 0) { value = default; return false; } + if (!Bundle.AllAssetsArrived()) { value = default; return false; } // Everything arrived, so every lease this bundle will ever hold is settled and it is safe to report - // a failure. Handles keep their own error, so this needs no bookkeeping of its own and reports in - // the order the assets were requested. isReady stays false, so every later call throws again. - List? failures = null; - foreach (var handle in Handles) - { - if (handle.Error is not null) - { - (failures ??= []).Add(handle.Error); - } - } - - if (failures != null) - { - if (failures.Count == 1) { throw failures[0]; } - throw new AggregateException(failures); - } + // a failure. isReady stays false, so every later call throws again. + Bundle.ThrowOnFailedAssets(); - result = value = Factory(new AssetHandleResolver(this)); + result = value = Factory(new AssetHandleResolver(Bundle)); isReady = true; return true; } diff --git a/source/CapriKit.AssetPipeline/AssetManager.cs b/source/CapriKit.AssetPipeline/AssetManager.cs index 29bec19..310c3f0 100644 --- a/source/CapriKit.AssetPipeline/AssetManager.cs +++ b/source/CapriKit.AssetPipeline/AssetManager.cs @@ -23,10 +23,9 @@ public sealed partial class AssetManager : IDisposable private readonly Lock RequestLock; private readonly Dictionary> Outstanding; - // Everything that holds, or is going to hold, leases. Kept so that Dispose can name whoever forgot + // Every bundle that holds, or is going to hold, leases. Kept so that Dispose can name whoever forgot // to unload instead of only reporting that some number of assets was left behind. private readonly HashSet LiveBundles; - private readonly HashSet UnbuiltBundles; public AssetManager(ILoggerFactory logger, ScopedFileSystem fileSystem) { @@ -39,7 +38,6 @@ public AssetManager(ILoggerFactory logger, ScopedFileSystem fileSystem) RequestLock = new(); Outstanding = []; LiveBundles = []; - UnbuiltBundles = []; } /// @@ -59,42 +57,23 @@ public void RegisterTranscoder(IAssetTranscoder - /// Use to defines a bundle of assets to load. The call site is captured so that a bundle that is never - /// unloaded can point at the code that created it, callers should not pass the two arguments themselves. + /// Creates a bundle to load assets into. The bundle owns everything loaded into it until it is + /// unloaded, so it is the thing to keep and to dispose. The call site is captured so that a bundle that + /// is never unloaded can point at the code that created it, callers should not pass those two arguments. /// Threading: thread-safe. /// - public AssetBundleBuilder CreateBundle([CallerFilePath] string file = "", [CallerLineNumber] int line = 0) + public AssetBundle CreateBundle([CallerFilePath] string file = "", [CallerLineNumber] int line = 0) { - return new AssetBundleBuilder(this, $"{Path.GetFileName(file.AsSpan())}:{line}"); - } + var bundle = new AssetBundle(this, $"{Path.GetFileName(file.AsSpan())}:{line}"); + lock (RequestLock) { LiveBundles.Add(bundle); } - /// - /// Registers a builder as a future owner of leases, see . - /// Threading: thread-safe. - /// - internal void RegisterBuilder(AssetBundleBuilder builder) - { - lock (RequestLock) { UnbuiltBundles.Add(builder); } - } - - /// - /// Hands ownership of the leases from the builder to the bundle it built. From here on the bundle is - /// what has to be unloaded, and what reports if that never happens. - /// Threading: thread-safe. - /// - internal void RegisterBundle(AssetBundleBuilder builder, AssetBundle bundle) - { - lock (RequestLock) - { - UnbuiltBundles.Remove(builder); - LiveBundles.Add(bundle); - } + return bundle; } /// /// Starts loading an asset. The asset will either be loaded from the cache, from disk, or rebuild and then loaded. /// The caller gets a handle to be used in an which can be resolved - /// to the actual asset when loading finishes using + /// to the actual asset when loading finishes using /// Threading: thread-safe, can be called from any thread concurrently. This method guarantees that the same asset /// is not loaded multiple times concurrently. /// @@ -174,7 +153,7 @@ private async Task RequestAsset(AssetId id, TSettings setting /// Threading: Unload updates the internal state of the bundle using a lock so that it is safe /// to unload the same bundle from multiple threads. /// - public void Unload(AssetBundle bundle) + internal void Unload(AssetBundle bundle) { try { @@ -204,7 +183,7 @@ public void Unload(AssetBundle bundle) /// Materializes assets that have finished loading, removes unused items from the cache /// and hot-reloads changed assets. /// A failed load does not throw here, it is handed to the handles that were waiting for it and - /// surfaces from instead. + /// surfaces from instead. /// Threading: Should only be called from the primary thread. /// public void Update() @@ -282,8 +261,8 @@ private void DrainIncoming() } /// - /// Names every bundle that still holds leases, and every builder whose assets never got a bundle to - /// belong to, so that the leak the throws about can be traced back to its code. + /// Names every bundle that still holds leases, so that the leak the throws + /// about can be traced back to the code that caused it. /// private void ReportAssetsThatWereNeverUnloaded() { @@ -291,16 +270,10 @@ private void ReportAssetsThatWereNeverUnloaded() { foreach (var bundle in LiveBundles) { - LogBundleNotUnloaded(Logger, bundle.Origin, bundle.Handles.Count); - } - - foreach (var builder in UnbuiltBundles) - { - // A builder that never loaded anything is simply unused, only one that did is a problem: - // its handles have no bundle to be unloaded through. - if (builder.RequestedAssets > 0) + // A bundle that never loaded anything is simply unused, not a leak. + if (bundle.Total > 0) { - LogBundleNeverBuilt(Logger, builder.Origin, builder.RequestedAssets); + LogBundleNotUnloaded(Logger, bundle.Origin, bundle.Total); } } } @@ -392,7 +365,4 @@ private IAssetTranscoder GetTranscoder() whe [LoggerMessage(Level = LogLevel.Error, Message = "The asset bundle created at {origin} was never unloaded, it still holds {assets} asset(s)")] private static partial void LogBundleNotUnloaded(ILogger logger, string origin, int assets); - - [LoggerMessage(Level = LogLevel.Error, Message = "The asset bundle builder created at {origin} loaded {assets} asset(s) but was never built, so those assets could never be unloaded")] - private static partial void LogBundleNeverBuilt(ILogger logger, string origin, int assets); } diff --git a/source/CapriKit.AssetPipeline/README.md b/source/CapriKit.AssetPipeline/README.md index 8e3da15..961e52c 100644 --- a/source/CapriKit.AssetPipeline/README.md +++ b/source/CapriKit.AssetPipeline/README.md @@ -15,15 +15,15 @@ assetManager.RegisterTranscoder(new VertexShaderTranscoder(GraphicsDevice)); record UIBundle(PixelShader Shader, Texture2D Font); ``` -3. Use the asset manager to create a bundle builder and start building (if needed) and loading assets. +3. Use the asset manager to create a bundle and start building (if needed) and loading assets into it. The bundle owns everything you load into it, so it is the thing to keep and to dispose. ``` -var builder = assetManager.CreateBundle(); -var shaderHandle = builder.Load(new AssetId("./shader.hlsl")); -var fontHandle = builder.Load(new AssetId("./"robo.png")); +using var bundle = assetManager.CreateBundle(); +var shaderHandle = bundle.Load(new AssetId("./shader.hlsl")); +var fontHandle = bundle.Load(new AssetId("./"robo.png")); ``` -4. Use the obtained handles to describe how the bundle can be constructed once building and loading finishes. +4. Use the obtained handles to describe how the contents of the bundle can be constructed once building and loading finishes. This gives you a loader, which only reports progress: dropping it costs nothing, unloading still goes through the bundle. ``` -using var bundle = builder.Build(r => new UIBundle(r.get(shaderHandle), r.get(fontHandle)); +var loader = bundle.Build(r => new UIBundle(r.get(shaderHandle), r.get(fontHandle)); ``` 5. Do not forget to update the asset manager each frame so that it can manage the loading tasks and other bookkeeping. @@ -33,7 +33,7 @@ assetManager.Update(); 6. Check each frame if loading finished and obtain your strongly typed contents ``` -if (bundle.isReady(out var ui)) +if (loader.isReady(out var ui)) { // Yay! } diff --git a/source/CapriKit.Tests/AssetPipeline/AssetManagerTests.cs b/source/CapriKit.Tests/AssetPipeline/AssetManagerTests.cs index 1c36da3..3c522a3 100644 --- a/source/CapriKit.Tests/AssetPipeline/AssetManagerTests.cs +++ b/source/CapriKit.Tests/AssetPipeline/AssetManagerTests.cs @@ -47,45 +47,45 @@ public async Task LoadAsset() var id = new AssetId(AssetFile); - var builder = assetManager.CreateBundle(); - var handle = builder.Load(id, default); - var loader = builder.Build(resolver => new TestBundle(resolver.Get(handle))); + var bundle = assetManager.CreateBundle(); + var handle = bundle.Load(id, default); + var loader = bundle.Build(resolver => new TestBundle(resolver.Get(handle))); - TestBundle? bundle = null; + TestBundle? contents = null; await Assert.That(() => { assetManager.Update(); - return loader.IsReady(out bundle); + return loader.IsReady(out contents); }) .Eventually(v => v.IsTrue(), TimeSpan.FromSeconds(5)); - await Assert.That(bundle).IsNotNull(); - await Assert.That(bundle.Asset.Text).IsEqualTo(TranscoderText); + await Assert.That(contents).IsNotNull(); + await Assert.That(contents.Asset.Text).IsEqualTo(TranscoderText); // Load again to verify loading the same thing twice gives us the cached value - var altBuilder = assetManager.CreateBundle(); - var altHandle = altBuilder.Load(id, default); - var altLoader = altBuilder.Build(resolver => new TestBundle(resolver.Get(altHandle))); + var altBundle = assetManager.CreateBundle(); + var altHandle = altBundle.Load(id, default); + var altLoader = altBundle.Build(resolver => new TestBundle(resolver.Get(altHandle))); - TestBundle? altBundle = null; + TestBundle? altContents = null; await Assert.That(() => { assetManager.Update(); - return altLoader.IsReady(out altBundle); + return altLoader.IsReady(out altContents); }) .Eventually(v => v.IsTrue(), TimeSpan.FromSeconds(5)); - await Assert.That(altBundle).IsNotNull(); - await Assert.That(altBundle.Asset).IsSameReferenceAs(bundle.Asset); + await Assert.That(altContents).IsNotNull(); + await Assert.That(altContents.Asset).IsSameReferenceAs(contents.Asset); // Both bundles lease the one shared instance, so it only really goes away once both gave it back - assetManager.Unload(loader); + bundle.Dispose(); assetManager.Update(); - await Assert.That(bundle.Asset.IsDisposed).IsFalse(); + await Assert.That(contents.Asset.IsDisposed).IsFalse(); - assetManager.Unload(altLoader); + altBundle.Dispose(); assetManager.Update(); - await Assert.That(bundle.Asset.IsDisposed).IsTrue(); + await Assert.That(contents.Asset.IsDisposed).IsTrue(); // Nothing is left over, so shutting down is quiet await Assert.That(() => assetManager.Dispose()).ThrowsNothing(); @@ -106,9 +106,9 @@ public async Task Unload_ReturnsTheLeaseOfAnAssetThatWasStillLoading() var assetManager = new AssetManager(NullLoggerFactory.Instance, fileSystem); assetManager.RegisterTranscoder(transcoder); - var builder = assetManager.CreateBundle(); - var handle = builder.Load(new AssetId(AssetFile)); - var bundle = builder.Build(resolver => new TestBundle(resolver.Get(handle))); + var bundle = assetManager.CreateBundle(); + var handle = bundle.Load(new AssetId(AssetFile)); + var loader = bundle.Build(resolver => new TestBundle(resolver.Get(handle))); // Act: unload before the first Update, so the asset is still loading and holds no lease yet. // Disposing the bundle is the same thing as unloading it, and is how this is meant to be written. @@ -122,8 +122,9 @@ await Assert.That(() => }) .Eventually(v => v.IsTrue(), TimeSpan.FromSeconds(5)); - // The bundle no longer owns its assets, so it refuses to hand them out rather than serving disposed ones - await Assert.That(() => bundle.IsReady(out _)).Throws(); + // The bundle no longer owns its assets, so its loader refuses to hand them out rather than + // serving disposed ones + await Assert.That(() => loader.IsReady(out _)).Throws(); await Assert.That(() => assetManager.Dispose()).ThrowsNothing(); } @@ -144,14 +145,14 @@ public async Task Dispose_ReportsBundlesThatWereNeverUnloaded() var assetManager = new AssetManager(logger, fileSystem); assetManager.RegisterTranscoder(new TrackingTextTranscoder()); - var builder = assetManager.CreateBundle(); - var handle = builder.Load(new AssetId(AssetFile)); - var bundle = builder.Build(resolver => new TestBundle(resolver.Get(handle))); + var bundle = assetManager.CreateBundle(); + var handle = bundle.Load(new AssetId(AssetFile)); + var loader = bundle.Build(resolver => new TestBundle(resolver.Get(handle))); await Assert.That(() => { assetManager.Update(); - return bundle.IsReady(out _); + return loader.IsReady(out _); }) .Eventually(v => v.IsTrue(), TimeSpan.FromSeconds(5)); @@ -187,9 +188,9 @@ public async Task Load_RetriesAnAssetWhoseFirstLoadFailed() // Act: the first load fails while building. Update stays quiet about it, the failure is handed to // the bundle that was waiting and surfaces from IsReady. - var failedBuilder = assetManager.CreateBundle(); - var failedHandle = failedBuilder.Load(id); - var failedLoader = failedBuilder.Build(resolver => new TestBundle(resolver.Get(failedHandle))); + var failedBundle = assetManager.CreateBundle(); + var failedHandle = failedBundle.Load(id); + var failedLoader = failedBundle.Build(resolver => new TestBundle(resolver.Get(failedHandle))); AssetLoadException? failure = null; await Assert.That(() => @@ -211,17 +212,17 @@ await Assert.That(() => // Act: take away the reason the build failed and ask for the very same asset again transcoder.ShouldFail = false; - var retryBuilder = assetManager.CreateBundle(); - var retryHandle = retryBuilder.Load(id); - var retryLoader = retryBuilder.Build(resolver => new TestBundle(resolver.Get(retryHandle))); + var retryBundle = assetManager.CreateBundle(); + var retryHandle = retryBundle.Load(id); + var retryLoader = retryBundle.Build(resolver => new TestBundle(resolver.Get(retryHandle))); // Control: a second, untouched asset requested at the same moment, to show that a failure never // stopped the manager as a whole and that the retry above is what actually changed. await fileSystem.WriteAllText(HealthyFile, TranscoderText); - var healthyBuilder = assetManager.CreateBundle(); - var healthyHandle = healthyBuilder.Load(new AssetId(HealthyFile)); - var healthyLoader = healthyBuilder.Build(resolver => new TestBundle(resolver.Get(healthyHandle))); + var healthyBundle = assetManager.CreateBundle(); + var healthyHandle = healthyBundle.Load(new AssetId(HealthyFile)); + var healthyLoader = healthyBundle.Build(resolver => new TestBundle(resolver.Get(healthyHandle))); await Assert.That(() => { @@ -247,11 +248,11 @@ await Assert.That(() => await Assert.That(() => failedLoader.IsReady(out _)).Throws(); // Every bundle can be unloaded, the failed one included: it returns nothing because its only handle - // never took a lease. Getting that wrong would return the lease that retryLoader holds on the same + // never took a lease. Getting that wrong would return the lease that retryBundle holds on the same // asset, so the clean dispose below is what proves the counting is right. - assetManager.Unload(failedLoader); - assetManager.Unload(retryLoader); - assetManager.Unload(healthyLoader); + assetManager.Unload(failedBundle); + assetManager.Unload(retryBundle); + assetManager.Unload(healthyBundle); await Assert.That(() => assetManager.Dispose()).ThrowsNothing(); } @@ -268,9 +269,9 @@ public async Task Load_ThrowsOnTheCallingThreadWhenNoTranscoderIsRegistered() await fileSystem.WriteAllText(AssetFile, TranscoderText); using var assetManager = new AssetManager(NullLoggerFactory.Instance, fileSystem); - var builder = assetManager.CreateBundle(); + var bundle = assetManager.CreateBundle(); - await Assert.That(() => builder.Load(new AssetId(AssetFile))).Throws(); + await Assert.That(() => bundle.Load(new AssetId(AssetFile))).Throws(); } /// @@ -289,23 +290,23 @@ public async Task Unload_ReturnsOneLeasePerHandle() var id = new AssetId(AssetFile); - var builder = assetManager.CreateBundle(); - var first = builder.Load(id); - var second = builder.Load(id); - var loader = builder.Build(resolver => new TwiceBundle(resolver.Get(first), resolver.Get(second))); + var bundle = assetManager.CreateBundle(); + var first = bundle.Load(id); + var second = bundle.Load(id); + var loader = bundle.Build(resolver => new TwiceBundle(resolver.Get(first), resolver.Get(second))); - TwiceBundle? bundle = null; + TwiceBundle? contents = null; await Assert.That(() => { assetManager.Update(); - return loader.IsReady(out bundle); + return loader.IsReady(out contents); }) .Eventually(v => v.IsTrue(), TimeSpan.FromSeconds(5)); // Both handles resolved to the single cached instance, but each of them took its own lease - await Assert.That(bundle!.Second).IsSameReferenceAs(bundle.First); + await Assert.That(contents!.Second).IsSameReferenceAs(contents.First); - assetManager.Unload(loader); + assetManager.Unload(bundle); await Assert.That(() => assetManager.Dispose()).ThrowsNothing(); } @@ -325,15 +326,15 @@ public async Task Update_DoesNotThrowWhenAHotReloadFails() var assetManager = new AssetManager(NullLoggerFactory.Instance, fileSystem); assetManager.RegisterTranscoder(transcoder); - var builder = assetManager.CreateBundle(); - var handle = builder.Load(new AssetId(AssetFile)); - var loader = builder.Build(resolver => new TestBundle(resolver.Get(handle))); + var bundle = assetManager.CreateBundle(); + var handle = bundle.Load(new AssetId(AssetFile)); + var loader = bundle.Build(resolver => new TestBundle(resolver.Get(handle))); - TestBundle? bundle = null; + TestBundle? contents = null; await Assert.That(() => { assetManager.Update(); - return loader.IsReady(out bundle); + return loader.IsReady(out contents); }) .Eventually(v => v.IsTrue(), TimeSpan.FromSeconds(5)); @@ -353,9 +354,9 @@ await Assert.That(() => // Assert: the failure stayed inside the hot-reload path and the asset kept its old contents await Assert.That(thrown).IsNull(); - await Assert.That(bundle!.Asset.Text).IsEqualTo(TranscoderText); + await Assert.That(contents!.Asset.Text).IsEqualTo(TranscoderText); - assetManager.Unload(loader); + assetManager.Unload(bundle); assetManager.Dispose(); } } From 7d394543b2492f63ef9d0c9dcc32e7b09a8b2d15 Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Thu, 27 Aug 2026 20:25:16 +0200 Subject: [PATCH 51/53] WIP --- CapriKit.slnx | 3 + source/CapriKit.AssetPipeline/Asset.cs | 13 +- source/CapriKit.AssetPipeline/AssetBundle.cs | 407 +++++++++--------- source/CapriKit.AssetPipeline/AssetManager.cs | 48 ++- .../CapriKit.AssetPipeline.csproj | 1 + .../Playground/AssetBundle.cs | 139 ++++++ .../CapriKit.Collections.csproj | 3 + source/CapriKit.Collections/OneOrMany.cs | 160 +++++++ .../{Async => Primitives}/JobResult.cs | 9 +- .../AssetPipeline/AssetManagerTests.cs | 49 ++- source/CapriKit.Tests/CapriKit.Tests.csproj | 1 + .../Collections/OneOrManyTests.cs | 125 ++++++ 12 files changed, 740 insertions(+), 218 deletions(-) create mode 100644 source/CapriKit.AssetPipeline/Playground/AssetBundle.cs create mode 100644 source/CapriKit.Collections/CapriKit.Collections.csproj create mode 100644 source/CapriKit.Collections/OneOrMany.cs rename source/CapriKit.Concurrency/{Async => Primitives}/JobResult.cs (84%) create mode 100644 source/CapriKit.Tests/Collections/OneOrManyTests.cs diff --git a/CapriKit.slnx b/CapriKit.slnx index 2851bfd..3c388c6 100644 --- a/CapriKit.slnx +++ b/CapriKit.slnx @@ -42,6 +42,9 @@ + + + diff --git a/source/CapriKit.AssetPipeline/Asset.cs b/source/CapriKit.AssetPipeline/Asset.cs index c93cb80..bd75762 100644 --- a/source/CapriKit.AssetPipeline/Asset.cs +++ b/source/CapriKit.AssetPipeline/Asset.cs @@ -20,10 +20,21 @@ public override string ToString() } } +/// +/// An asset +/// +internal abstract record Asset(AssetId Id); + +/// +/// An asset, including its identifier +/// +internal record Asset(AssetId Id, TAsset Value) : Asset(Id) + where TAsset : class; + /// /// An asset, including its identifier and information on how it was built. /// -internal record Asset(AssetId Id, TAsset Value, AssetBuildMetaData BuildMetaData) +internal sealed record Asset(AssetId Id, TAsset Value, AssetBuildMetaData BuildMetaData) : Asset(Id, Value) where TAsset : class; /// diff --git a/source/CapriKit.AssetPipeline/AssetBundle.cs b/source/CapriKit.AssetPipeline/AssetBundle.cs index d94ed12..45cc8dd 100644 --- a/source/CapriKit.AssetPipeline/AssetBundle.cs +++ b/source/CapriKit.AssetPipeline/AssetBundle.cs @@ -1,203 +1,206 @@ -using System.Diagnostics.CodeAnalysis; - -namespace CapriKit.AssetPipeline; - -/// -/// A set of assets that are loaded together and unloaded together. The bundle holds one lease per asset it -/// loaded for as long as it lives, disposing it hands every one of them back. Use to -/// describe the strongly typed value those assets add up to, the bundle itself owns their lifetime. -/// Threading: create, load and build from one thread, usually the primary one. -/// -public sealed class AssetBundle : IDisposable -{ - private readonly List HandleList = []; - private readonly AssetManager Manager; - - internal AssetBundle(AssetManager manager, string origin) - { - Manager = manager; - Origin = origin; - } - - /// - /// The number of assets in this bundle. - /// - public int Total => HandleList.Count; - - /// - /// The number of assets that have arrived, successfully or not, updated every time - /// is called. - /// - public int Loaded { get; private set; } - - /// - /// The last asset that had arrived the last time was - /// called, in the order the assets were requested. Meant to put a name on a loading screen, not to - /// report the exact order in which assets finished. - /// - public AssetId? LastCompletedItem { get; private set; } - - /// - /// Every handle in this bundle, failed ones included, kept for as long as the bundle lives. Unloading - /// counts handles rather than distinct assets because the pool hands out one lease per resolved handle: - /// an asset this bundle asked for twice holds two leases, and one that failed to load holds none. - /// - internal IReadOnlyList Handles => HandleList; - - internal bool IsActive { get; set; } = true; - - /// - /// The file and line that created this bundle, used to name it if it is never unloaded. - /// - internal string Origin { get; } - - /// - /// Starts building (if needed) and loading an asset, and adds it to this bundle. The handle it returns - /// is used in the lambda for to - /// describe how the assets in this bundle add up to one strongly typed value. Load everything the - /// bundle needs before building it, an asset loaded afterwards still belongs to the bundle but the - /// loaders that were already built cannot see it. - /// - public AssetHandle Load(AssetId id, TSettings settings) - where TAsset : class - { - var handle = Manager.Load(id, settings); - handle.Owner = this; - HandleList.Add(handle); - return handle; - } - - /// - public AssetHandle Load(AssetId id) - where TAsset : class - => Load(id, default); - - /// - /// Creates the loader that reports the progress of this bundle and that builds the strongly typed value - /// its assets add up to. The loader borrows the bundle, it does not own it: this bundle stays the thing - /// that has to be unloaded, which is why building twice is harmless. - /// - public AssetBundleLoader Build(Func factory) - where TBundle : notnull - => new(this, factory); - - /// - /// Unloads this bundle, see . - /// Unloading a bundle twice is safe, the second call does nothing. - /// - public void Dispose() => Manager.Unload(this); - - /// - /// Updates and and reports whether every asset in - /// the bundle arrived, successfully or not. Walks all handles rather than keeping a list of the pending - /// ones: a bundle holds a handful of assets, and one list is easier to reason about than two that have - /// to agree with each other. - /// - internal bool AllAssetsArrived() - { - var arrived = 0; - foreach (var handle in HandleList) - { - // An asset that failed has arrived too, it just arrived as an error instead of as a value. - if (handle.IsCompleted) - { - arrived++; - LastCompletedItem = handle.Id; - } - } - - Loaded = arrived; - return arrived == HandleList.Count; - } - - /// - /// Reports the assets in this bundle that could not be built or loaded, in the order they were - /// requested. Only call this once is true: until then an asset might - /// still take a lease, and reporting a failure before that would leave it unaccounted for. - /// - internal void ThrowOnFailedAssets() - { - List? failures = null; - foreach (var handle in HandleList) - { - if (handle.Error is not null) - { - (failures ??= []).Add(handle.Error); - } - } - - if (failures is null) { return; } - if (failures.Count == 1) { throw failures[0]; } - - throw new AggregateException(failures); - } -} - -/// -/// Tracks the loading progress of an and builds the strongly typed value its -/// assets add up to once they all arrived. Purely a view on the bundle: unloading goes through the bundle, -/// so a loader that is dropped costs nothing. -/// -public sealed class AssetBundleLoader - where TBundle : notnull -{ - private readonly AssetBundle Bundle; - private readonly Func Factory; - - private TBundle? result; - private bool isReady; - - internal AssetBundleLoader(AssetBundle bundle, Func factory) - { - Bundle = bundle; - Factory = factory; - } - - /// - public int Total => Bundle.Total; - - /// - public int Loaded => Bundle.Loaded; - - /// - public AssetId? LastCompletedItem => Bundle.LastCompletedItem; - - /// - /// Checks whether the bundle finished loading, and if so builds and returns it. The result is - /// cached so calling IsReady multiple times after loading finished is OK. - /// Throws a when an asset in this bundle could not be built or - /// loaded. Throws an if multiple assets failed to load. - /// Throws an once the bundle has been unloaded. - /// - public bool IsReady([NotNullWhen(true)] out TBundle? value) - { - // An unloaded bundle gave its leases back, so the assets it would hand out may already be disposed. - ObjectDisposedException.ThrowIf(!Bundle.IsActive, Bundle); - - if (isReady) { value = result!; return true; } - - if (!Bundle.AllAssetsArrived()) { value = default; return false; } - - // Everything arrived, so every lease this bundle will ever hold is settled and it is safe to report - // a failure. isReady stays false, so every later call throws again. - Bundle.ThrowOnFailedAssets(); - - result = value = Factory(new AssetHandleResolver(Bundle)); - isReady = true; - return true; - } - - /// - /// Busy waits until the bundle finishes loading, then builds and returns it. - /// Threading: primary thread only. - /// - public TBundle WaitUntilReady() - { - var wait = new SpinWait(); - while (true) - { - if (IsReady(out var bundle)) { return bundle; } - wait.SpinOnce(); - } - } +using System.Diagnostics.CodeAnalysis; + +namespace CapriKit.AssetPipeline; + +/// +/// A set of assets that are loaded together and unloaded together. The bundle holds one lease per asset it +/// loaded for as long as it lives, disposing it hands every one of them back. Use to +/// describe the strongly typed value those assets add up to, the bundle itself owns their lifetime. +/// Threading: create, load and build from one thread, usually the primary one. +/// +public sealed class AssetBundle : IDisposable +{ + private readonly List HandleList = []; + private readonly AssetManager Manager; + + internal AssetBundle(AssetManager manager, string origin) + { + Manager = manager; + Origin = origin; + } + + /// + /// The number of assets in this bundle. + /// + public int Total => HandleList.Count; + + /// + /// The number of assets that have arrived, successfully or not, updated every time + /// is called. + /// + public int Loaded { get; private set; } + + /// + /// The last asset that had arrived the last time was + /// called, in the order the assets were requested. Meant to put a name on a loading screen, not to + /// report the exact order in which assets finished. + /// + public AssetId? LastCompletedItem { get; private set; } + + /// + /// Every handle in this bundle, failed ones included, kept for as long as the bundle lives. Unloading + /// counts handles rather than distinct assets because the pool hands out one lease per resolved handle: + /// an asset this bundle asked for twice holds two leases, and one that failed to load holds none. + /// + internal IReadOnlyList Handles => HandleList; + + internal bool IsActive { get; set; } = true; + + /// + /// The file and line that created this bundle, used to name it if it is never unloaded. + /// + internal string Origin { get; } + + /// + /// Starts building (if needed) and loading an asset, and adds it to this bundle. The handle it returns + /// is used in the lambda for to + /// describe how the assets in this bundle add up to one strongly typed value. Load everything the + /// bundle needs before building it, an asset loaded afterwards still belongs to the bundle but the + /// loaders that were already built cannot see it. + /// + public AssetHandle Load(AssetId id, TSettings settings) + where TAsset : class + { + var handle = Manager.Load(id, settings); + handle.Owner = this; + HandleList.Add(handle); + return handle; + } + + /// + public AssetHandle Load(AssetId id) + where TAsset : class + => Load(id, default); + + /// + /// Creates the loader that reports the progress of this bundle and that builds the strongly typed value + /// its assets add up to. The loader borrows the bundle, it does not own it: this bundle stays the thing + /// that has to be unloaded, which is why building twice is harmless. + /// + public AssetBundleLoader Build(Func factory) + where TBundle : notnull + => new(this, factory); + + /// + /// Unloads this bundle, see . + /// Unloading a bundle twice is safe, the second call does nothing. + /// + public void Dispose() => Manager.Unload(this); + + /// + /// Updates and and reports whether every asset in + /// the bundle arrived, successfully or not. Walks all handles rather than keeping a list of the pending + /// ones: a bundle holds a handful of assets, and one list is easier to reason about than two that have + /// to agree with each other. + /// + internal bool AllAssetsArrived() + { + var arrived = 0; + foreach (var handle in HandleList) + { + // An asset that failed has arrived too, it just arrived as an error instead of as a value. + if (handle.IsCompleted) + { + arrived++; + LastCompletedItem = handle.Id; + } + } + + Loaded = arrived; + return arrived == HandleList.Count; + } + + /// + /// Reports the assets in this bundle that could not be built or loaded, in the order they were + /// requested. Only call this once is true: until then an asset might + /// still take a lease, and reporting a failure before that would leave it unaccounted for. + /// + internal void ThrowOnFailedAssets() + { + List? failures = null; + foreach (var handle in HandleList) + { + if (handle.Error is not null) + { + (failures ??= []).Add(handle.Error); + } + } + + if (failures is null) { return; } + if (failures.Count == 1) { throw failures[0]; } + + throw new AggregateException(failures); + } +} + +/// +/// Tracks the loading progress of an and builds the strongly typed value its +/// assets add up to once they all arrived. Purely a view on the bundle: unloading goes through the bundle, +/// so a loader that is dropped costs nothing. +/// +public sealed class AssetBundleLoader + where TBundle : notnull +{ + private readonly AssetBundle Bundle; + private readonly Func Factory; + + private TBundle? result; + private bool isReady; + + internal AssetBundleLoader(AssetBundle bundle, Func factory) + { + Bundle = bundle; + Factory = factory; + } + + /// + public int Total => Bundle.Total; + + /// + public int Loaded => Bundle.Loaded; + + /// + public AssetId? LastCompletedItem => Bundle.LastCompletedItem; + + /// + /// Checks whether the bundle finished loading, and if so builds and returns it. The result is + /// cached so calling IsReady multiple times after loading finished is OK. + /// Throws a when an asset in this bundle could not be built or + /// loaded. Throws an if multiple assets failed to load. + /// Throws an once the bundle has been unloaded. + /// + public bool IsReady([NotNullWhen(true)] out TBundle? value) + { + // An unloaded bundle gave its leases back, so the assets it would hand out may already be disposed. + ObjectDisposedException.ThrowIf(!Bundle.IsActive, Bundle); + + if (isReady) { value = result!; return true; } + + if (!Bundle.AllAssetsArrived()) { value = default; return false; } + + // Everything arrived, so every lease this bundle will ever hold is settled and it is safe to report + // a failure. isReady stays false, so every later call throws again. + Bundle.ThrowOnFailedAssets(); + + result = value = Factory(new AssetHandleResolver(Bundle)); + isReady = true; + return true; + } + + /// + /// Busy waits until the bundle finishes loading, then builds and returns it. + /// Threading: primary thread only. + /// WARNING: assets only arrive when runs, and that is the primary + /// thread's job too, so this spins forever unless every asset in the bundle came straight from the + /// cache. Do not call it until it either pumps the manager itself or is removed. + /// + public TBundle WaitUntilReady() + { + var wait = new SpinWait(); + while (true) + { + if (IsReady(out var bundle)) { return bundle; } + wait.SpinOnce(); + } + } } diff --git a/source/CapriKit.AssetPipeline/AssetManager.cs b/source/CapriKit.AssetPipeline/AssetManager.cs index 310c3f0..f3aa483 100644 --- a/source/CapriKit.AssetPipeline/AssetManager.cs +++ b/source/CapriKit.AssetPipeline/AssetManager.cs @@ -74,8 +74,9 @@ public AssetBundle CreateBundle([CallerFilePath] string file = "", [CallerLineNu /// Starts loading an asset. The asset will either be loaded from the cache, from disk, or rebuild and then loaded. /// The caller gets a handle to be used in an which can be resolved /// to the actual asset when loading finishes using - /// Threading: thread-safe, can be called from any thread concurrently. This method guarantees that the same asset - /// is not loaded multiple times concurrently. + /// Threading: the manager's own state is safe to touch from any thread concurrently, and this method + /// guarantees that the same asset is not loaded multiple times concurrently. The bundle above it is + /// what limits callers to one thread, see . /// internal AssetHandle Load(AssetId id, TSettings settings) where TAsset : class @@ -150,8 +151,10 @@ private async Task RequestAsset(AssetId id, TSettings setting /// that did load are returned, and the ones that failed never took anything that needs returning. So can /// a bundle that is still loading, the leases its assets take when they do arrive are returned right away. /// Unloading the same bundle twice is safe, the second call does nothing. - /// Threading: Unload updates the internal state of the bundle using a lock so that it is safe - /// to unload the same bundle from multiple threads. + /// Threading: the lock makes unloading safe against and against a second unload of + /// the same bundle, including from another thread. It does not make unloading safe against + /// on that same bundle, which touches the same list + /// without the lock. /// internal void Unload(AssetBundle bundle) { @@ -183,7 +186,9 @@ internal void Unload(AssetBundle bundle) /// Materializes assets that have finished loading, removes unused items from the cache /// and hot-reloads changed assets. /// A failed load does not throw here, it is handed to the handles that were waiting for it and - /// surfaces from instead. + /// surfaces from instead. Update can still throw for + /// reasons that are not about one asset failing to build: an asset id used for two different asset + /// types, or an asset whose own Dispose throws while the pool cleans up. /// Threading: Should only be called from the primary thread. /// public void Update() @@ -199,6 +204,7 @@ public void Update() // either made this list, or it misses and its own Load starts a fresh request. if (!Outstanding.Remove(result.Id, out var waiting)) { continue; } + var abandoned = 0; foreach (var handle in waiting) { if (result.Failure is not null) { handle.Fail(result.Failure); } @@ -207,12 +213,18 @@ public void Update() handle.Resolve(result.Materialize!()); // The bundle was unloaded while this asset was still on its way. Unload could not - // return the lease that resolving just took, because back then there was nothing to - // return yet, so give it back here instead. Materializing first and returning after - // is deliberate: it routes the asset through the pool, which is what disposes it. - if (handle.Owner is { IsActive: false }) { Cache.Return(handle.Id); } + // return the lease that resolving just took, because back then there was nothing + // to return yet, so this is where it has to be given back. + if (handle.Owner is { IsActive: false }) { abandoned++; } } } + + // Counted first and returned after, because resolving takes one lease per handle and the + // pool evicts at zero: returning as we went would drop an asset that two handles of the + // same bundle asked for to zero in between, and queue it for disposal twice. + // Returning rather than skipping the materialize is deliberate too, it routes the asset + // through the pool, which is what disposes it. + for (var i = 0; i < abandoned; i++) { Cache.Return(result.Id); } } } @@ -228,13 +240,22 @@ public void Update() /// public void Dispose() { + // Every step below runs even if an earlier one threw. This is the last chance to hand assets back + // to the pool and to name the bundles that were left behind, so one failing step must not take the + // others with it: swallowing here costs a log line, skipping would cost the whole diagnostic. + // Dispose the hot-reload manager first so that it can release any reference to // assets it might still hold. - HotReloadManager.Dispose(); + try { HotReloadManager.Dispose(); } + catch (Exception ex) { LogShutdownStepFailed(Logger, nameof(HotReloadManager), ex); } - DrainIncoming(); - ReportAssetsThatWereNeverUnloaded(); + try { DrainIncoming(); } + catch (Exception ex) { LogShutdownStepFailed(Logger, nameof(DrainIncoming), ex); } + try { ReportAssetsThatWereNeverUnloaded(); } + catch (Exception ex) { LogShutdownStepFailed(Logger, nameof(ReportAssetsThatWereNeverUnloaded), ex); } + + // Deliberately not guarded: the leak it throws about is the whole point of the check. Cache.Dispose(); } @@ -365,4 +386,7 @@ private IAssetTranscoder GetTranscoder() whe [LoggerMessage(Level = LogLevel.Error, Message = "The asset bundle created at {origin} was never unloaded, it still holds {assets} asset(s)")] private static partial void LogBundleNotUnloaded(ILogger logger, string origin, int assets); + + [LoggerMessage(Level = LogLevel.Error, Message = "Shutting the asset manager down failed during {step}, the remaining steps still ran")] + private static partial void LogShutdownStepFailed(ILogger logger, string step, Exception exception); } diff --git a/source/CapriKit.AssetPipeline/CapriKit.AssetPipeline.csproj b/source/CapriKit.AssetPipeline/CapriKit.AssetPipeline.csproj index 3c2e557..0a2fe4c 100644 --- a/source/CapriKit.AssetPipeline/CapriKit.AssetPipeline.csproj +++ b/source/CapriKit.AssetPipeline/CapriKit.AssetPipeline.csproj @@ -4,6 +4,7 @@ + diff --git a/source/CapriKit.AssetPipeline/Playground/AssetBundle.cs b/source/CapriKit.AssetPipeline/Playground/AssetBundle.cs new file mode 100644 index 0000000..7c2f3cb --- /dev/null +++ b/source/CapriKit.AssetPipeline/Playground/AssetBundle.cs @@ -0,0 +1,139 @@ +using CapriKit.Concurrency.Primitives; +using System.Diagnostics.CodeAnalysis; + +namespace CapriKit.AssetPipeline.Playground; + + +public record Example(string SomeAsset) +{ + public static void ExampleLoad(AssetManager manager) + { + using var bundle = new AssetBundle(manager); + var handle = bundle.Request(new AssetId("text.txt"), default); + bundle.Build(r => new Example(r.Get(handle))); + + while (true) + { + if (bundle.IsReady(out var content)) + { + // We can now access our bundle contents + return; + } + } + } +} + + +public sealed record AssetHandle(AssetId Id); + + +public sealed class AssetBundle(AssetManager assetManager) : IDisposable + where TContent : class +{ + private readonly AssetManager AssetManager = assetManager; + private Func? resolver; + private readonly Lock Lock = new(); + private readonly HashSet Requested = []; + private readonly Dictionary> Received = []; + private bool isReady; + private TContent? content; + public bool IsDisposed { get; private set; } + + public void Build(Func resolver) + { + this.resolver = resolver; + } + + /// + /// Threading: thread-safe + /// + internal bool Accept(AssetId id, JobResult result) + { + lock (Lock) + { + if (IsDisposed) { return false; } + Received[id] = result; + return true; + } + } + + internal TAsset Get(AssetHandle handle) + where TAsset : class + { + if (handle.Owner != this) + { + throw new InvalidOperationException("Attempted to resolve a handle that was not created by this bundle"); + } + + return ((Asset)Received[handle.Id].GetOrThrow()).Value; + } + + /// + /// Threading: unsafe, only one thread can access this method at the same time, but it is safe for other threads + /// to access the other methods on this type. + /// + public AssetHandle Request(AssetId id, TSettings settings) + where TAsset : class + { + if (!Requested.Add(id)) { throw new Exception($"You cannot request the same asset twice from the same bundle: {id}."); } + var handle = AssetManager.Load(id, settings); + handle.Owner = this; + return handle; + } + + /// + /// Threading: unsafe, only one thread can access this method at the same time, but it is safe for other threads + /// to access the other methods on this type. + /// + public bool IsReady([NotNullWhen(true)] out TContent? content) + { + if (isReady) + { + content = this.content!; + return true; + } + + if (resolver == null) + { + throw new InvalidOperationException("Call Build before calling IsReady"); + } + + + if (Received.Count == Requested.Count) + { + content = resolver(new AssetResolver(this)); + isReady = true; + return true; + } + + content = default; + return false; + } + + public void Dispose() + { + lock (Lock) + { + if (IsDisposed) { return; } + + foreach (var kv in Received) + { + AssetManager.Unload(kv.Key); + } + Received.Clear(); + IsDisposed = true; + } + } + + public sealed class AssetResolver + { + private readonly AssetBundle Owner; + internal AssetResolver(AssetBundle owner) + { + Owner = owner; + } + + public TAsset Get(AssetHandle handle) + where TAsset : class => Owner.Get(handle); + } +} diff --git a/source/CapriKit.Collections/CapriKit.Collections.csproj b/source/CapriKit.Collections/CapriKit.Collections.csproj new file mode 100644 index 0000000..c369e11 --- /dev/null +++ b/source/CapriKit.Collections/CapriKit.Collections.csproj @@ -0,0 +1,3 @@ + + + diff --git a/source/CapriKit.Collections/OneOrMany.cs b/source/CapriKit.Collections/OneOrMany.cs new file mode 100644 index 0000000..7b1ab59 --- /dev/null +++ b/source/CapriKit.Collections/OneOrMany.cs @@ -0,0 +1,160 @@ +namespace CapriKit.Collections; + +/// +/// A small mutable collection optimized for storing a single value. +/// Use when the collection usually holds many values, when you need the elements +/// to be contiguous, or when you need to pass the values around as an . +/// +/// +/// This is a mutable struct, so dictionary[key].Add(value) silently mutates a copy and +/// throws the value away. Take a reference to the value inside the dictionary instead: +/// +/// ref var values = ref CollectionsMarshal.GetValueRefOrAddDefault(dictionary, key, out _); +/// values.Add(value); +/// +/// Any insert into that dictionary can resize it and invalidates such a reference, so use it +/// immediately and never store it. +/// +public struct OneOrMany +{ + private T? head; + private T[]? tail; + + /// Creates a collection holding a single value + public OneOrMany(T value) + { + head = value; + tail = null; + Count = 1; + } + + /// The number of values in the collection + public int Count { get; private set; } + + /// Gets or replaces the value at , in insertion order. + /// If the index is negative or not below . + public T this[int index] + { + readonly get + { + ThrowIfOutOfRange(index); + return index == 0 ? head! : tail![index - 1]; + } + set + { + ThrowIfOutOfRange(index); + if (index == 0) { head = value; } + else { tail![index - 1] = value; } + } + } + + /// Appends a value. + public void Add(T value) + { + if (Count == 0) + { + head = value; + } + else + { + var slot = Count - 1; + if (tail is null) { tail = new T[2]; } + else if (slot == tail.Length) { Array.Resize(ref tail, tail.Length * 2); } + tail[slot] = value; + } + + Count++; + } + + /// Returns the index of the first value equal to , or -1. + public readonly int IndexOf(T value) + { + var comparer = EqualityComparer.Default; + for (var i = 0; i < Count; i++) + { + if (comparer.Equals(this[i], value)) + { + return i; + } + } + + return -1; + } + + /// Returns whether any value equals . + public readonly bool Contains(T value) => IndexOf(value) >= 0; + + /// Removes the first value equal to and reports whether it was found. + public bool Remove(T value) + { + var index = IndexOf(value); + if (index < 0) + { + return false; + } + + RemoveAt(index); + return true; + } + + /// Removes the value at , shifting the values after it down. + /// If the index is negative or not below . + public void RemoveAt(int index) + { + ThrowIfOutOfRange(index); + + for (var i = index; i < Count - 1; i++) + { + this[i] = this[i + 1]; + } + + Count--; + + // Overwrite the slot that just fell outside the collection, otherwise the removed value stays + // reachable from this struct and the garbage collector cannot free it + if (Count == 0) { head = default; } + else { tail![Count - 1] = default!; } + } + + /// Removes all values and releases the overflow array, so a later second value allocates again. + public void Clear() + { + head = default; + tail = null; + Count = 0; + } + + /// + /// Returns a struct enumerator, so foreach over this collection does not allocate. + /// The enumerator works on a copy: values added or removed while it runs are not observed. + /// + /// + /// This type deliberately does not implement . Enumerating through + /// that interface (or through LINQ) boxes the struct, which is the allocation this type exists + /// to avoid. Copy the values into a list yourself if you really need an . + /// + public readonly Enumerator GetEnumerator() => new(this); + + private readonly void ThrowIfOutOfRange(int index) + { + ArgumentOutOfRangeException.ThrowIfNegative(index); + ArgumentOutOfRangeException.ThrowIfGreaterThanOrEqual(index, Count); + } + + /// Enumerates the values of a in insertion order. + public struct Enumerator + { + private readonly OneOrMany values; + private int index; + + internal Enumerator(OneOrMany source) + { + values = source; + index = -1; + } + + public readonly T Current => values[index]; + + public bool MoveNext() => ++index < values.Count; + } +} diff --git a/source/CapriKit.Concurrency/Async/JobResult.cs b/source/CapriKit.Concurrency/Primitives/JobResult.cs similarity index 84% rename from source/CapriKit.Concurrency/Async/JobResult.cs rename to source/CapriKit.Concurrency/Primitives/JobResult.cs index 8c0ca30..c775406 100644 --- a/source/CapriKit.Concurrency/Async/JobResult.cs +++ b/source/CapriKit.Concurrency/Primitives/JobResult.cs @@ -1,7 +1,8 @@ using System.Diagnostics; using System.Runtime.ExceptionServices; -namespace CapriKit.Concurrency.Async; +namespace CapriKit.Concurrency.Primitives; + public sealed class JobResult { private readonly string Id; @@ -39,4 +40,10 @@ public void Match(Action onSuccess, Action await Assert.That(() => assetManager.Dispose()).ThrowsNothing(); } + /// + /// The corner case of : a bundle that + /// asked for the same asset twice takes two leases when it arrives, so it has to give back two. Giving + /// them back one at a time as each handle resolves drops the count to zero in between, which evicts and + /// queues the asset for disposal once per handle and double frees it. + /// + [Test] + public async Task Unload_DisposesAnAssetLoadedTwiceOnceWhenItWasStillLoading() + { + var fileSystem = new InMemoryFileSystem().ScopedTo("C:/Test"); + await fileSystem.WriteAllText(AssetFile, TranscoderText); + + var transcoder = new TrackingTextTranscoder(); + var assetManager = new AssetManager(NullLoggerFactory.Instance, fileSystem); + assetManager.RegisterTranscoder(transcoder); + + var id = new AssetId(AssetFile); + + var bundle = assetManager.CreateBundle(); + var first = bundle.Load(id); + var second = bundle.Load(id); + _ = bundle.Build(resolver => new TwiceBundle(resolver.Get(first), resolver.Get(second))); + + // Act: unload before the first Update, so both handles are still on their way + bundle.Dispose(); + + await Assert.That(() => + { + assetManager.Update(); + return transcoder.Decoded.Count == 1 && transcoder.Decoded.All(asset => asset.IsDisposed); + }) + .Eventually(v => v.IsTrue(), TimeSpan.FromSeconds(5)); + + // Assert: the one asset behind both handles was disposed exactly once + await Assert.That(transcoder.Decoded.Single().DisposeCount).IsEqualTo(1); + + await Assert.That(() => assetManager.Dispose()).ThrowsNothing(); + } + /// /// Everything that is loaded has to be unloaded before the game quits. The pool notices when that did /// not happen but can only count the assets left behind, so the manager names the bundle they belong to @@ -370,9 +409,15 @@ internal sealed class TextAsset(string text) : IDisposable public string Text { get; set; } = text; /// Lets tests see whether the pool really let go of this asset. - public bool IsDisposed { get; private set; } + public bool IsDisposed => DisposeCount > 0; + + /// + /// Counted rather than flagged: an asset that is disposed twice would double free the native + /// resources of a real asset, so a test has to be able to tell one from two. + /// + public int DisposeCount { get; private set; } - public void Dispose() => IsDisposed = true; + public void Dispose() => DisposeCount++; } /// diff --git a/source/CapriKit.Tests/CapriKit.Tests.csproj b/source/CapriKit.Tests/CapriKit.Tests.csproj index 78a40ca..1d3bf87 100644 --- a/source/CapriKit.Tests/CapriKit.Tests.csproj +++ b/source/CapriKit.Tests/CapriKit.Tests.csproj @@ -18,6 +18,7 @@ + diff --git a/source/CapriKit.Tests/Collections/OneOrManyTests.cs b/source/CapriKit.Tests/Collections/OneOrManyTests.cs new file mode 100644 index 0000000..a942caa --- /dev/null +++ b/source/CapriKit.Tests/Collections/OneOrManyTests.cs @@ -0,0 +1,125 @@ +using System.Runtime.InteropServices; +using CapriKit.Collections; + +namespace CapriKit.Tests.Collections; + +internal class OneOrManyTests +{ + [Test] + public async Task Add() + { + var values = new OneOrMany("a"); + + values.Add("b"); + values.Add("c"); + + await Assert.That(values.Count).IsEqualTo(3); + await Assert.That(values[0]).IsEqualTo("a"); + await Assert.That(values[1]).IsEqualTo("b"); + await Assert.That(values[2]).IsEqualTo("c"); + } + + [Test] + public async Task Add_GrowsBeyondInitialTailCapacity() + { + var values = new OneOrMany(); + + // The tail array starts at two elements, so this forces it to grow twice + for (var i = 0; i < 8; i++) + { + values.Add(i); + } + + await Assert.That(values.Count).IsEqualTo(8); + for (var i = 0; i < 8; i++) + { + await Assert.That(values[i]).IsEqualTo(i); + } + } + + [Test] + public async Task Indexer_OutOfRange() + { + var values = new OneOrMany("a"); + + await Assert.That(() => values[1]).Throws(); + await Assert.That(() => values[-1]).Throws(); + } + + [Test] + public async Task Remove() + { + var values = new OneOrMany("a"); + values.Add("b"); + values.Add("c"); + + var removed = values.Remove("b"); + + await Assert.That(removed).IsTrue(); + await Assert.That(values.Count).IsEqualTo(2); + await Assert.That(values[0]).IsEqualTo("a"); + await Assert.That(values[1]).IsEqualTo("c"); + await Assert.That(values.Contains("b")).IsFalse(); + } + + [Test] + public async Task Remove_Head() + { + var values = new OneOrMany("a"); + values.Add("b"); + + // Removing the inline value has to promote the first value of the tail into it + var removed = values.Remove("a"); + + await Assert.That(removed).IsTrue(); + await Assert.That(values.Count).IsEqualTo(1); + await Assert.That(values[0]).IsEqualTo("b"); + } + + [Test] + public async Task Clear() + { + var values = new OneOrMany("a"); + values.Add("b"); + + values.Clear(); + + await Assert.That(values.Count).IsEqualTo(0); + await Assert.That(values.Contains("a")).IsFalse(); + } + + [Test] + public async Task GetEnumerator() + { + var values = new OneOrMany("a"); + values.Add("b"); + values.Add("c"); + + var enumerated = new List(); + foreach (var value in values) + { + enumerated.Add(value); + } + + await Assert.That(enumerated).IsEquivalentTo(new List { "a", "b", "c" }); + } + + [Test] + public async Task UsedAsDictionaryValue() + { + var map = new Dictionary>(); + + // The pattern this type is meant for: mutate the value in place, without copying it out + foreach (var (key, value) in new[] { ("odd", 1), ("even", 2), ("odd", 3) }) + { + ref var values = ref CollectionsMarshal.GetValueRefOrAddDefault(map, key, out _); + values.Add(value); + } + + await Assert.That(map["odd"].Count).IsEqualTo(2); + await Assert.That(map["odd"][0]).IsEqualTo(1); + await Assert.That(map["odd"][1]).IsEqualTo(3); + await Assert.That(map["even"].Count).IsEqualTo(1); + await Assert.That(map["even"][0]).IsEqualTo(2); + } +} From 60eb2cd5429161d94f897cb10811a8f4c6291760 Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Thu, 27 Aug 2026 21:28:09 +0200 Subject: [PATCH 52/53] WIP-m --- research/Bundle-owned bookkeeping.html | 1044 +++++++++++++++++ .../Playground/AssetBundle.cs | 123 +- 2 files changed, 1118 insertions(+), 49 deletions(-) create mode 100644 research/Bundle-owned bookkeeping.html diff --git a/research/Bundle-owned bookkeeping.html b/research/Bundle-owned bookkeeping.html new file mode 100644 index 0000000..75bd74b --- /dev/null +++ b/research/Bundle-owned bookkeeping.html @@ -0,0 +1,1044 @@ +Bundle-Owned Bookkeeping + + + + + + +
+ +
+
+ CapriKit.AssetPipeline + feature/asset_pipeline + 2026-08-27 +
+

Bundle-Owned Bookkeeping

+

+ The Playground sketch moves the record of outstanding work out of AssetHandle + and into AssetBundle<TContent>. This is what that actually buys, what it does not, + and the five things in the sketch that would bite. +

+
+ +
+ + + +
+ +
+
+

+ Yes — it simplifies materially. But the simplification is two independent decisions bundled + together, and only one of them needs the rewrite. +

+

+ One lease per distinct asset (the HashSet<AssetId> Requested) + is what deletes the nastiest corner cases in the current code. You can have that today, in about + five lines, without touching the architecture. +

+

+ Pushing results into the bundle instead of polling handles is the part that needs + the rewrite, and it earns it: readiness becomes O(1), progress becomes exact, and the + unloaded-while-loading handoff collapses from three cooperating flags into one atomic decision + under one lock. +

+

+ The trouble is concentrated in one place: the bundle has to be known to + AssetManager.Load before the request starts. Attach it afterwards, as the + sketch does, and cached assets never arrive at all. +

+
+
+ +
+

Where the bookkeeping lives today

+

+ Seven stores, and the invariant that ties them together is not written down in any one of them: + the pool hands out one lease per resolved handle, not per distinct asset. +

+ +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
StoreOwnerJob
OutstandingAssetManagerAssetId → List<AssetHandle>. Collapses concurrent requests for one id into one build.
value / error / isResolvedAssetHandleThe cross-thread publication point. volatile write on isResolved publishes the other two.
OwnerAssetHandleBack-pointer so Update can ask whether the bundle is still alive.
HandleListAssetBundleThe lease ledger — one entry per Load call, not per distinct asset.
IsActiveAssetBundleMarks a bundle unloaded so late arrivals give their lease straight back.
LiveBundlesAssetManagerNames whoever forgot to unload, at shutdown.
EntriesAssetPoolThe actual refcounts.
+
+ +

+ The cost of that spread is visible in AssetManager.Update, where a single question — + did this asset arrive at a bundle that still wants it? — is answered by consulting three of + those stores in sequence, and where the order of two loops is load-bearing: +

+ + 
// Counted first and returned after, because resolving takes one lease per handle and the
+// pool evicts at zero: returning as we went would drop an asset that two handles of the
+// same bundle asked for to zero in between, and queue it for disposal twice.
+for (var i = 0; i < abandoned; i++) { Cache.Return(result.Id); }
+ +

+ Two of the eight tests in AssetManagerTests exist only to pin that comment down: + Unload_ReturnsOneLeasePerHandle and + Unload_DisposesAnAssetLoadedTwiceOnceWhenItWasStillLoading. +

+
+ +
+

What the shift actually is

+

+ Not "the bundle holds a list instead of the handle". The unit of bookkeeping changes from + the handle to the asset id, scoped to a bundle — and delivery + flips from pull to push. +

+ +
+
+ Today · handle-owned +

The handle is the rendezvous

+
    +
  • AssetHandle is mutable and published across threads.
    • +
  • Update writes into each handle; the bundle finds out by walking them.
    • +
  • Readiness is an O(n) scan, once per bundle per frame.
    • +
  • A bundle's lease count equals its handle count.
    • +
  • The bundle is attached to the handle after the request starts.
    • +
+
+ +
+
+ +
+

What it genuinely removes

+ +
+ +
+ Win +
+

One lease per asset kills the count-then-return dance

+

+ Requested is a set and Request throws on a duplicate id, so a bundle + can never hold two leases on one asset. Unload becomes a walk over + Received, the abandoned counter disappears, and both of the + corner-case tests above stop describing anything real. +

+

+ It also makes an existing implicit rule explicit: Outstanding and + AssetPool.Entries are already keyed by AssetId alone, so the same id + with two different settings objects is already unsupported. The sketch just says so out loud. +

+

+ Separable. AssetBundle.Load could keep a Dictionary<AssetId, + AssetHandle> and hand back the existing handle on a repeat id. That is a five-line + change to one method and it banks this entire win today, independent of everything else on + this page. +

+
+
+ +
+ Win +
+

Readiness becomes O(1), and progress becomes exact and free

+

+ Total = Requested.Count, Loaded = Received.Count. + LastCompletedItem is one assignment inside Accept. This retires + Polling AssetBundleLoader IsReady outright: proposals A, B, and C all exist to make a + per-frame scan cheaper, and a pushed result makes the scan disappear. +

+

+ Worth noting why that note rejected the push model, because the sketch answers it. + Both reasons — the cache hit that resolves before Owner is assigned, and the race + between assigning Owner and Resolve — are consequences of attaching + the bundle after the fact. Passing the bundle into Load removes the + category of problem, not just the two instances. +

+
+
+ +
+ Win +
+

AssetHandle stops being a concurrency primitive

+

+ No volatile, no Resolve/Fail, no back-pointer, no + release-ordering comment to maintain. All of the memory-ordering reasoning currently spread + across AssetHandle, AssetManager.Update and + AssetBundleLoader.IsReady collapses into one lock in one type. This is the part + you were after, and it holds up. +

+
+
+ +
+ Win +
+

The unloaded-while-loading handoff becomes one atomic decision

+

+ Today the answer is assembled from bundle.IsActive, + handle.IsLoaded and the abandoned tally. In the sketch, + Accept returning false under the bundle's own lock is the + decision: +

+ 
lock (Lock)
+{
+    if (IsDisposed) { return false; }   // manager returns the lease
+    Received[id] = result;                // Dispose will return the lease
+    return true;
+}
+

+ Either the result lands in Received and Dispose hands it back, or it + is refused and the manager hands it back. Never both, never neither — and + Dispose takes the same lock, so there is no window between the two. That is a + materially tighter invariant than what is there now, and it is the piece that cannot be + retrofitted without the rewrite. +

+
+
+ +
+
+ +
+

What it does not remove

+ +
+ +
+ Wash +
+

Outstanding stays, it just changes what it holds

+

+ It becomes Dictionary<AssetId, List<AssetBundle>>. Same shape, same + job — collapsing concurrent requests for one id into one build. Nothing is saved here. + (It is, incidentally, a textbook OneOrMany<AssetBundle>: almost always one + bundle waiting, occasionally several.) +

+
+
+ +
+ Wash +
+

Two types in, two types out

+

+ AssetBundle<TContent> cannot sit in LiveBundles or receive + Accept polymorphically without a non-generic base or interface. So you merge + AssetBundle + AssetBundleLoader<TBundle> into one type, then + split it again into AssetBundle + AssetBundle<TContent>. +

+

+ What does improve is the caller's side: one object to hold and dispose instead of a + bundle plus a loader. The README's step 4 ("this gives you a loader") would need rewriting. +

+
+
+ +
+ Wash +
+

Requesting after building is still unguarded

+

+ Received.Count == Requested.Count is exactly as fragile as today's + arrived == HandleList.Count if a caller interleaves requests with readiness + checks — request A, A arrives, check readiness, and the bundle latches complete before B is + ever asked for. Today that is handled by a sentence in the XML docs. +

+

+ The new shape at least has an obvious place to fix it: Build is already the + "I am done asking" moment, so let it seal the bundle and make Request throw + afterwards. Cheap, and it converts a documented convention into a checked one. +

+
+
+ +
+ No impact +
+

Hot reload does not care

+

+ HotReloadManager, TrackedAsset, AssetEncoder, + AssetDecoder, IAssetTranscoder and AssetPool work off + the pool and their own maps and never look at a handle or a bundle. The scariest code in the + project is entirely outside the blast radius — worth knowing before committing to this. +

+
+
+ +
+
+ +
+

What breaks in the sketch as written

+

+ Four of these are fixable in a line or two; the first one is the reason the whole design needs + AssetManager.Load to change shape. +

+ +
+ +
+ Fatal +
+

Cached assets never arrive

+ 
var handle = AssetManager.Load<TAsset, TSettings>(id, settings);
+handle.Owner = this;   // too late
+

+ AssetManager.Load resolves a cache hit inside its RequestLock + section, before this line runs. Nothing is ever written into Received, so + Received.Count never reaches Requested.Count and the bundle waits + forever — for precisely the assets that were cheapest to get. +

+

+ The fix is the structural change the whole approach rests on: + Load(bundle, id, settings), with the cache-hit branch calling + bundle.Accept(id, JobResult.Success(...)) while it still holds the lock. That also + removes the need for Owner entirely — the guard in Get becomes + Requested.Contains(handle.Id), and the handle stays immutable. +

+
+
+ +
+ Bug +
+

Dispose returns leases that were never taken

+ 
foreach (var kv in Received)
+{
+    AssetManager.Unload(kv.Key);   // including the failures
+}
+

+ A failed JobResult is in Received — correctly, it arrived, + it just arrived as an error — but it never took a lease. AssetPool.Return throws + InvalidOperationException("Returned {id} which was not found in the cache.") on + it. Today's Unload guards this with if (handle.IsLoaded); the new + one needs the equivalent check on the result. +

+
+
+ +
+ Bug +
+

The manager is never told the bundle is gone

+

+ Dispose hands leases back by id but nothing removes the bundle from + LiveBundles. The shutdown leak report would then name bundles that were + unloaded properly — which is worse than no report, because it burns the one diagnostic that + makes a real leak findable. +

+
+
+ +
+ Race +
+

IsReady reads the dictionary that Accept writes

+

+ Accept is documented thread-safe and takes the lock; IsReady and + Get read Received.Count and Received[handle.Id] with no + lock at all. If those really can run on different threads, that is an unsynchronized read of a + Dictionary mid-Add. +

+

+ Pick one and commit: either results are delivered on the main thread from + Update — which is what the pipeline does today, and then the lock in + Accept buys nothing and should go — or delivery really is concurrent and + IsReady/Get have to take the lock too. +

+
+
+ +
+ Regression +
+

The first failure hides the rest

+

+ Get calls JobResult.GetOrThrow() from inside the caller's factory + lambda, so the first broken asset throws and the other failures in the bundle are never + reported. Today ThrowOnFailedAssets collects them all and throws one + AggregateException before the factory runs. +

+

+ Two smaller things ride along: GetOrThrow rethrows the transcoder's raw exception, + losing the AssetLoadException wrapper that names the asset (wrap it before + storing), and JobResult<T> keys by string, so + AssetId gets stringified on the way in. +

+
+
+ +
+
+ +
+

The one threading rule to write down

+

+ A second lock is the real risk in this design, and it is not hypothetical — the sketch is one + method body away from it. +

+ + 
// AssetBundle.Dispose
+lock (Lock)                       // takes BundleLock ...
+{
+    AssetManager.Unload(kv.Key);   // ... then RequestLock
+}
+
+// AssetManager.Update
+lock (RequestLock)                // takes RequestLock ...
+{
+    bundle.Accept(id, result);     // ... then BundleLock
+}
+ +

+ Today Unload must take RequestLock — it touches LiveBundles. + The moment Dispose also has to tell the manager the bundle is gone (bug 3 above), the + two paths acquire the same pair of locks in opposite orders, and the game thread disposing a bundle + deadlocks against the main thread in Update. +

+

+ Rule: never hold the bundle lock while calling into the manager. Collect the ids + under the lock, release it, then return them. Better still — while every result is delivered from + Update on the main thread, do not introduce the bundle lock at all. A lock only ever + taken by one thread is pure cost plus this hazard. +

+
+ +
+

Blast radius

+ +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
FileChangeWhy
AssetHandle.csGuttedBecomes a record of an id. AssetHandleResolver moves inside the bundle.
AssetBundle.csRewrittenNon-generic base plus AssetBundle<TContent>; AssetBundleLoader deleted.
AssetManager.cs3 methodsLoad takes the bundle, Update pushes, Unload gets simpler.
Asset.csTrimmedLoadResult can become JobResult<Asset>.
AssetManagerTests.csAll 8Every test drives the bundle/loader pair that no longer exists.
README.mdSteps 4 & 6Build no longer hands back a loader.
AssetPool.csUntouchedStill the only thing that counts references.
HotReloadManager.cs, TrackedAsset.csUntouchedWorks off the pool; never sees a handle or a bundle.
AssetEncoder, AssetDecoder, IAssetTranscoderUntouchedBelow the ownership layer entirely.
+
+ +

+ Two of the eight tests are deleted rather than ported: both duplicate-asset corner cases + stop being expressible. That is the clearest single measurement of the simplification. +

+
+ +
+

Recommended path

+

+ Ordered so each stage is independently useful and independently revertable — the first one is worth + doing whether or not the rest ever happens. +

+ +
    +
  1. + 01 +
    + ~5 lines · no architecture change +

    Dedupe by id in AssetBundle.Load

    +

    + Keep a Dictionary<AssetId, AssetHandle> and return the existing handle on a + repeat. One lease per asset, so Unload walks ids, the abandoned + count-then-return loop in Update goes away, and two corner-case tests retire. + This is most of the bookkeeping win, today. +

    +
    +
    2. + +
  2. + 02 +
    + The rewrite · 3 files +

    Pass the bundle into Load and push results

    +

    + Outstanding becomes AssetId → bundles; the cache-hit branch calls + Accept under RequestLock; Update materializes once per + accepting bundle and returns the lease for any that refuses. Gut + AssetHandle last — it falls out on its own once nothing writes to it. +

    +

    + Keep delivery on the main thread. No bundle lock, no lock-order rule to remember. +

    +
    +
    3. + +
  3. + 03 +
    + Polish +

    Seal the bundle at Build

    +

    + Make Request throw after Build, and collect every failure in a + pre-pass before invoking the factory. Turns two documented conventions into checked ones. +

    +
    +
    4. +
+ +

+ What would change this recommendation: if bundles ever need to be built from a worker thread while + the main thread pumps Update, the bundle lock becomes real, and stage 2 should land + with the lock-order rule from above written into the XML docs on Accept and + Dispose — not discovered later. +

+
+ +
+ Sources read: AssetManager.cs · AssetBundle.cs · AssetHandle.cs · AssetPool.cs · Asset.cs · + HotReloadManager.cs · TrackedAsset.cs · AssetManagerTests.cs · Playground/AssetBundle.cs · + Concurrency/Primitives/{JobResult,LightweightChannel}.cs
+ Prior notes: “Polling AssetBundleLoader IsReady” (2026-08-20) · “Recovering from a failed asset load” (2026-08-25)
+ The Playground sketch does not compile as written; the four compiler errors are symptoms of bugs 1 and 3. +
+ +
+
+
diff --git a/source/CapriKit.AssetPipeline/Playground/AssetBundle.cs b/source/CapriKit.AssetPipeline/Playground/AssetBundle.cs index 7c2f3cb..83f5edf 100644 --- a/source/CapriKit.AssetPipeline/Playground/AssetBundle.cs +++ b/source/CapriKit.AssetPipeline/Playground/AssetBundle.cs @@ -8,9 +8,9 @@ public record Example(string SomeAsset) { public static void ExampleLoad(AssetManager manager) { - using var bundle = new AssetBundle(manager); - var handle = bundle.Request(new AssetId("text.txt"), default); - bundle.Build(r => new Example(r.Get(handle))); + var builder = new AssetBundleBuilder(manager); + var handle = builder.Request(new AssetId("text.txt"), default); + using var bundle = builder.Build(r => new Example(r.Get(handle))); while (true) { @@ -24,67 +24,91 @@ public static void ExampleLoad(AssetManager manager) } -public sealed record AssetHandle(AssetId Id); +public readonly record struct AssetHandle(AssetId Id, Guid Owner); +public sealed class AssetBundleBuilder(AssetManager assetManager) + where TContents : class +{ + private readonly HashSet Requested = []; + private readonly List>> Requests = []; + private readonly Guid Id = Guid.NewGuid(); + + public AssetHandle Request(AssetId id, TSettings settings) + where TAsset : class + { + if (!Requested.Add(id)) { throw new Exception($"You cannot request the same asset twice for the same bundle: {id}."); } + Requests.Add(b => assetManager.Load(id, settings, b)); + return new AssetHandle(id, Id); + } + + public AssetBundle Build(Func.AssetResolver, TContents> resolver) + { + var bundle = new AssetBundle(Id, assetManager, Requested, resolver); + foreach (var request in Requests) + { + request(bundle); + } -public sealed class AssetBundle(AssetManager assetManager) : IDisposable + return bundle; + } +} + +public interface IAssetRequester +{ + /// + /// Takes ownership of the asset (or failure). + /// + /// True if ownership is accepted. False if the requester stopped accepting new inputs. + public bool Accept(AssetId id, JobResult result); +} + +public sealed class AssetBundle : IAssetRequester, IDisposable where TContent : class { - private readonly AssetManager AssetManager = assetManager; - private Func? resolver; + private readonly Guid Id; + private readonly AssetManager AssetManager; + private readonly Func Resolver; private readonly Lock Lock = new(); - private readonly HashSet Requested = []; - private readonly Dictionary> Received = []; + private readonly IReadOnlySet Requested; + private readonly Dictionary> Received; private bool isReady; private TContent? content; - public bool IsDisposed { get; private set; } - public void Build(Func resolver) + internal AssetBundle(Guid id, AssetManager assetManager, IReadOnlySet requested, Func resolver) { - this.resolver = resolver; + Id = id; + AssetManager = assetManager; + Requested = requested; + Resolver = resolver; + Received = []; } + public bool IsDisposed { get; private set; } + /// - /// Threading: thread-safe + /// Threading: unsafe, only one thread can access this method at the same time. /// - internal bool Accept(AssetId id, JobResult result) + public bool Accept(AssetId id, JobResult result) { - lock (Lock) - { - if (IsDisposed) { return false; } - Received[id] = result; - return true; - } + if (IsDisposed) { return false; } + Received[id] = result; + return true; } internal TAsset Get(AssetHandle handle) where TAsset : class { - if (handle.Owner != this) + if (handle.Owner != Id) { - throw new InvalidOperationException("Attempted to resolve a handle that was not created by this bundle"); + throw new InvalidOperationException("Attempted to resolve a handle that was not created for this bundle"); } return ((Asset)Received[handle.Id].GetOrThrow()).Value; } /// - /// Threading: unsafe, only one thread can access this method at the same time, but it is safe for other threads - /// to access the other methods on this type. - /// - public AssetHandle Request(AssetId id, TSettings settings) - where TAsset : class - { - if (!Requested.Add(id)) { throw new Exception($"You cannot request the same asset twice from the same bundle: {id}."); } - var handle = AssetManager.Load(id, settings); - handle.Owner = this; - return handle; - } - - /// - /// Threading: unsafe, only one thread can access this method at the same time, but it is safe for other threads - /// to access the other methods on this type. - /// + /// Threading: unsafe, only one thread can access this method at the same time. + /// public bool IsReady([NotNullWhen(true)] out TContent? content) { if (isReady) @@ -93,7 +117,7 @@ public bool IsReady([NotNullWhen(true)] out TContent? content) return true; } - if (resolver == null) + if (Resolver == null) { throw new InvalidOperationException("Call Build before calling IsReady"); } @@ -101,7 +125,7 @@ public bool IsReady([NotNullWhen(true)] out TContent? content) if (Received.Count == Requested.Count) { - content = resolver(new AssetResolver(this)); + content = Resolver(new AssetResolver(this)); isReady = true; return true; } @@ -110,19 +134,20 @@ public bool IsReady([NotNullWhen(true)] out TContent? content) return false; } + /// + /// Threading: unsafe, only one thread can access this method at the same time. + /// public void Dispose() { - lock (Lock) - { - if (IsDisposed) { return; } + if (IsDisposed) { return; } - foreach (var kv in Received) - { - AssetManager.Unload(kv.Key); - } - Received.Clear(); - IsDisposed = true; + foreach (var kv in Received) + { + // Ignore any assets that failed to load during disposal + kv.Value.Match((id, asset) => AssetManager.Unload(id), (id, ex) => { }); } + Received.Clear(); + IsDisposed = true; } public sealed class AssetResolver From 265edfda49167aeb585a3c60aa59af464c678e29 Mon Sep 17 00:00:00 2001 From: Roy Triesscheijn Date: Thu, 27 Aug 2026 22:30:35 +0200 Subject: [PATCH 53/53] foo --- .../Bundle-owned bookkeeping-with-loader.html | 887 ++++++++++++++++++ 1 file changed, 887 insertions(+) create mode 100644 research/Bundle-owned bookkeeping-with-loader.html diff --git a/research/Bundle-owned bookkeeping-with-loader.html b/research/Bundle-owned bookkeeping-with-loader.html new file mode 100644 index 0000000..c359001 --- /dev/null +++ b/research/Bundle-owned bookkeeping-with-loader.html @@ -0,0 +1,887 @@ +The Builder Split + + + + + + +
+ +
+
+ CapriKit.AssetPipeline + feature/asset_pipeline + Revision 2 + 2026-08-27 +
+

The Builder Split

+

+ The sketch now separates AssetBundleBuilder<TContents> from + AssetBundle<TContent> and defers every load until Build. That change + closes most of what the first pass flagged — by construction rather than by convention. Here is what is + left. +

+
+ +
+ + + +
+ +
+
+

+ This is the right shape. The builder is not extra ceremony — it is the thing that makes + Received.Count == Requested.Count a sound question to ask. +

+

+ Deferring the loads means the request set is complete before the first result can arrive, + and the bundle exists before the first Load call. Those two facts are what the + previous round's worst finding and its shakiest convention were both about, and both are now true + by construction rather than by documentation. +

+

+ What is left is small: one real logic bug the compiler is already pointing at, two type mismatches, + and one genuinely new failure mode that deferring the loads introduced — a throwing + Build now strands a bundle nobody can dispose. +

+
+
+ +
+

Scorecard against the first pass

+ +
+
3Fixed
+
2No longer applies
+
2Still open
+
4New
+
+ +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Finding from revision 1StatusWhy
Cached assets never arriveFixedThe bundle is constructed first and passed into Load; nothing starts before it exists.
Dispose returns leases for failuresFixedMatch only unloads on the success branch. (New bug rides along — see N2.)
Unsynchronized read of ReceivedGoneEverything is documented single-threaded now. The Lock field is dead — delete it.
Lock-order inversion between bundle and managerGoneNo second lock, so no ordering rule to get wrong.
Requesting after building is unguardedFixedRequest only exists on the builder. Residual: the builder can be reused — see N4.
Manager is never told the bundle is goneOpenDispose still returns leases by id and says nothing about itself.
First failure hides the restOpenGetOrThrow still fires from inside the caller's factory lambda.
+
+ +

The compiler agrees the direction is right — four errors became two, and both survivors are informative:

+ + 
error CS1501: No overload for method 'Load' takes 3 arguments            ← the manager change, not yet made
+error CS1503: Argument 1: cannot convert from 'string' to 'AssetBundle'  ← N2 below
+warning CS0649: Field 'AssetBundle<TContent>.content' is never assigned  ← N1 below
+
+ +
+

The mechanism that does the work

+

+ Worth stating plainly, because it is the whole design in six lines. Request records a + closure instead of starting a load, and Build replays them into a bundle that already + exists: +

+ + 
// AssetBundleBuilder
+Requests.Add(b => assetManager.Load<TAsset, TSettings>(id, settings, b));
+return new AssetHandle<TAsset>(id, Id);
+
+// ... later, in Build
+var bundle = new AssetBundle<TContents>(Id, assetManager, Requested, resolver);
+foreach (var request in Requests) { request(bundle); }
+return bundle;
+ +

+ The closure is what keeps Request<TAsset, TSettings> generic while + Requests stays a plain List<Action<…>> — the standard trick, + and the right one here. It costs one delegate plus one closure per asset, which the handle turning + into a struct pays back. +

+ +
+
+ Revision 1 +

Bundle attached after the fact

+
    +
  • Request starts the load, then sets Owner.
    • +
  • Cache hits resolve before the bundle is known.
    • +
  • The request set grows while results arrive.
    • +
  • Handle is a sealed record — one allocation each.
    • +
+
+ +
+ +

+ That re-entrancy is load-bearing and non-obvious: a fully cached bundle is ready the instant + Build returns, without a single Update. It works because Build + does not touch Received while iterating. Say so in a comment, or someone will later add + a readiness check to Build and be very confused. +

+
+ +
+

What the split newly buys

+ +
+ +
+ Win +
+

Abandoning a builder is free

+

+ Requesting and then never calling Build starts nothing, leases nothing, and leaks + nothing. Today the equivalent mistake — CreateBundle, Load, then + forget to Build — leaves a bundle holding leases that only the shutdown leak + report will ever mention. The new shape makes the harmless thing the default. +

+
+
+ +
+ Win +
+

IAssetRequester decouples the manager from bundles entirely

+

+ Outstanding becomes Dictionary<AssetId, List<IAssetRequester>> + and the manager stops knowing what a bundle is. That is a bigger deal than it looks: anything + that wants assets delivered — a streaming system, a level-of-detail loader, an editor preview + — can implement three lines of interface instead of pretending to be a bundle. +

+

+ Decide deliberately whether that extension point is public. Accept hands out + leases; a third-party implementation that returns true and then drops the result + leaks an asset with no diagnostic. Internal for now is the conservative call. +

+
+
+ +
+ Win +
+

The handle is now free, and the Guid is the reason

+

+ readonly record struct AssetHandle<TValue>(AssetId Id, Guid Owner) is 24 + bytes, copyable, with value equality and no allocation. The Guid is what makes + that possible: a reference to AssetBundle<TContent> would drag the generic + parameter into the handle, and even via IAssetRequester it would keep the bundle + alive and make the handle a lifetime-bearing thing. An opaque token keeps it a pure value. +

+

+ Pleasant side effect: default(AssetHandle<T>) carries + Guid.Empty, so it fails the owner check and throws rather than doing something + surprising. +

+
+
+ +
+
+ +
+

New findings

+ +
+ +
+ N1 · Bug +
+

The cached result is never stored, so the second IsReady returns null

+ 
if (Received.Count == Requested.Count)
+{
+    content = Resolver(new AssetResolver(this));  // assigns the out parameter
+    isReady = true;                              // this.content is still null
+    return true;
+}
+

+ The next call takes the isReady branch and hands back this.content!, + which was never written — null, through an [NotNullWhen(true)] + contract. This is exactly what CS0649 is reporting. Fix is + this.content = content = Resolver(…);. +

+

+ The polling loop in Example.ExampleLoad returns on the first success, so nothing + in the sketch exercises the second call. A game loop that keeps asking will. +

+
+
+ +
+ N2 · Bug +
+

Match hands you a string, not an AssetId

+ 
kv.Value.Match((id, asset) => AssetManager.Unload(id), (id, ex) => { });
+//              ^^ JobResult<T>.Id is a string, from the Job/BackgroundWorker world
+

+ Use kv.Key and the problem disappears. But it is worth reading as a signal: + JobResult<T> was built for BackgroundWorker, where jobs are + keyed by name. Here the dictionary already carries the real key, so the id inside the result + is dead weight that stringifies an AssetId on the way in. +

+

+ Either ignore JobResult's id entirely and always key off kv.Key, or + widen it to JobResult<TKey, TValue>. Reusing the type is right; inheriting + its key type is not. +

+
+
+ +
+ N3 · New failure mode +
+

A throwing Build strands a bundle nobody can dispose

+ 
foreach (var request in Requests)
+{
+    request(bundle);   // #3 throws (no transcoder) — #1 and #2 already leased
+}
+return bundle;        // never reached; the caller has no reference to dispose
+

+ This one is created by deferring the loads. Today + Load_ThrowsOnTheCallingThreadWhenNoTranscoderIsRegistered covers exactly this + case, and today the caller still holds the bundle and can dispose it. In the new shape the + bundle only exists inside Build. +

+

+ Fix is three lines — catch, bundle.Dispose(), rethrow — and it composes correctly + with everything else: the assets already leased are returned, and the ones still in flight hit + a disposed bundle whose Accept returns false, so the manager returns + those leases too. Worth a test of its own. +

+
+
+ +
+ N4 · Aliasing +
+

Requested is shared, not copied

+

+ The bundle holds an IReadOnlySet<AssetId> pointing at the builder's live + HashSet. Call Request after Build and the bundle's + total silently grows for an asset that will never be delivered — the bundle hangs, and + Get throws KeyNotFoundException if the counts ever line up again. + Building twice compounds it: both bundles share one set and one owner + Guid, so handles validate against either. +

+

+ Make Build consume the builder: copy the set ([.. Requested], or + ToFrozenSet since it is read-only from then on) and throw from + Request and Build afterwards. +

+
+
+ +
+ N5 · Tidy +
+

Two loose ends

+

+ private readonly Lock Lock = new(); is never taken — delete it, it advertises a + threading guarantee the XML docs explicitly withdraw. +

+

+ And new AssetBundleBuilder<Example>(manager) loses the + [CallerFilePath]/[CallerLineNumber] capture that + CreateBundle does today, which is what lets + Dispose_ReportsBundlesThatWereNeverUnloaded name the offending line. Primary + constructors take caller-info parameters fine — put them on the builder and keep the + new. +

+
+
+ +
+
+ +
+

Still carried from the first pass

+ +
+ +
+ Open +
+

Nothing tells the manager the bundle is gone

+

+ Dispose returns leases by id, so the pool balances — but the manager's live-bundle + registry never loses the entry, and the shutdown report starts naming bundles that were + unloaded correctly. That burns the one diagnostic that makes a real leak findable. +

+

+ Cleanest fix given the new shape: one call instead of a loop — + AssetManager.Unload(IAssetRequester requester, IEnumerable<AssetId> loaded). + The manager returns the leases and drops the registry entry together, and + Unload(AssetId) never has to exist as a callable thing that could corrupt a + refcount. +

+
+
+ +
+ Open +
+

The first failure still hides the rest

+

+ Get calls GetOrThrow() from inside the caller's factory lambda. + Today ThrowOnFailedAssets collects every failure and throws one + AggregateException before the factory runs. A pre-pass over Received + in IsReady restores that and puts the stack trace back in the pipeline instead of + in user code. +

+

+ Also unchanged: GetOrThrow rethrows the transcoder's raw exception, so the + AssetLoadException that names the asset has to be applied by the manager before + the result is stored. +

+
+
+ +
+ Decide +
+

What travels inside JobResult<object>

+

+ Get casts to the internal Asset<TAsset> wrapper and reads + .Value, but TrackAndTakeLease currently returns the canonical asset + itself. Deliver the raw value and cast (TAsset) — one less internal type crossing + a public-shaped interface, and the wrapper is only needed by the hot-reload tracking that + already builds its own. +

+
+
+ +
+
+ +
+

What the manager still owes

+

+ Unchanged from the first pass in substance, but the interface makes it smaller than it was. +

+ +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
MemberShapeNote
LoadLoad<TAsset, TSettings>(AssetId, TSettings, IAssetRequester)Cache hit calls Accept while still holding RequestLock.
OutstandingDictionary<AssetId, List<IAssetRequester>>A natural OneOrMany<IAssetRequester> — usually one waiter.
Updatematerialize per acceptorOne lease each; Accept returning false means return it.
UnloadUnload(IAssetRequester, IEnumerable<AssetId>)Leases and registry in one call. Keep it internal.
leak registryDictionary<IAssetRequester, string>Origin string, captured by the builder's caller-info parameters.
+
+ +

+ Everything below the ownership layer is still untouched: AssetPool, + HotReloadManager, TrackedAsset, AssetEncoder, + AssetDecoder, IAssetTranscoder. That has not changed between revisions and + remains the strongest practical argument for doing this at all. +

+
+ +
+

Revised path

+ +
    +
  1. + 01 +
    + Sketch only · ~10 lines +

    Close N1 through N5 in the Playground

    +

    + Assign this.content; swap Match's id for kv.Key; + wrap Build's replay in try/dispose/rethrow; copy Requested and + consume the builder; delete the dead Lock; move caller-info onto the builder + constructor. None of these need the manager to change, so the sketch can be made + self-consistent before anything real moves. +

    +
    +
    2. + +
  2. + 02 +
    + The one real change · AssetManager +

    Add the three-argument Load and push through IAssetRequester

    +

    + This is the single error the compiler is still reporting, and everything else in the design + already assumes it. Do the leak-registry call (Unload(requester, ids)) in the same + pass — it is the same method touching the same dictionary, and splitting it into two commits + just means shipping a broken diagnostic in between. +

    +
    +
    3. + +
  3. + 03 +
    + Behaviour parity +

    Restore aggregate failure reporting, then port the tests

    +

    + Pre-pass over Received before invoking the resolver, wrapping in + AssetLoadException at the manager. Then the eight tests in + AssetManagerTests: six port, two — both duplicate-asset corner cases — get + deleted, and two get added for N3 and for the second IsReady call. +

    +
    +
    4. +
+ +

+ One thing to keep an eye on that no fix addresses: the threading contract has narrowed. Today + AssetManager.Unload claims to be safe against Update from another thread. + The new bundle is single-threaded throughout, so disposal must happen on the game thread between + Update calls. That is almost certainly what every caller already does — and today's + claim is shaky anyway, since AssetBundle.Load mutates its handle list outside the lock + — but it is a documented guarantee being withdrawn, so withdraw it explicitly rather than quietly. +

+
+ + + +
+
+