From 6d66dc5c9674e1116c567de2f1a5e6b4d0b15f4b Mon Sep 17 00:00:00 2001 From: zxcheergo Date: Mon, 2 Nov 2026 23:12:34 +0800 Subject: [PATCH] HDR output for a profile that names its tone mapping pass (Breath of the Wild) On a screen with HDR, a Switch game is still its SDR picture: the game tone maps its HDR scene into 0..1 before it presents. A profile can now name its game's tone mapping pixel shader and an MSL file that replaces it (game.tsv `hdr `, DM_GAME_HDR*). The replacement keeps the translated shader's bindings and writes the game's result unchanged; beside it, into a host side texture, it writes how much brighter each pixel gets tone mapped up to the screen's headroom, and the SDR luminance it wrote. The present snapshots carry the highlights: the copy into a ring slot is a draw (dm/gfx/msl/hdr_merge.metal) that keeps the game's 8-bit picture and writes the excess, into the slot's RG11B10Float texture, wherever the picture still has the luminance the tone mapping wrote, so the HUD, fades and later effects drawn over the 3D picture take none. The present scales the 8-bit picture as before (bilinear, or MetalFX in its perceptual mode), decodes it and adds the highlights into an extended-range layer (RGBA16Float, extended linear sRGB). The peak follows the headroom the front end reports four times a second, eased; frames without the tone mapping pass and a peak of 1 take the plain copy. Copying into half float slots instead and letting MetalFX scale those made it run its HDR mode: 4.1 ms a present on an iPhone Duo (1915x1344 to 2853x2007) against about 2 ms on 8 bits, and heavier scenes fell to 30 fps. Breath of the Wild's pass (uking_pass_shader, f7ee6fe10b925543) is written out in hdr_tonemap.metal with the curve's top raised: the game's curve up to 0.6, then an exponential shoulder with the game's slope there rising toward the peak instead of 1. Switches: launcher setting HDR (DM_HDR, on where offered; the iOS app turns it off on a screen without EDR headroom), DM_HDR_PEAK, DM_HDR_MAX, DM_HDR_KNEE, DM_HDR_GAIN, DM_HDR_SHOT_RAW, and DM_CTL hdrpeak / hdrknee on the Mac. An explicit DM_PRESENT_SNAP=0 turns HDR output off, since the snapshots' copy carries the highlights (t_link_ticks sets it). Checked on an M1 Mac (macOS 27.0.1), the game built on this base with the NSP script (--allow-other-metal), headless, peak 4, screenshots of the extended-range picture through MetalFX at 2560x1440: the highlights on fire, torches and lanterns in Kakariko Village at dusk, none on the HUD, an NPC's dialog or the inventory's panels; at peak 1 no highlights. The host and Metal graphics tests pass. The same change on an older base ran on an iPhone Duo (iOS 27.1): headroom 8 at 50% brightness. Co-Authored-By: Claude Opus 5.5 --- dm/games/01007EF00011E000/README.md | 8 + dm/games/01007EF00011E000/game.tsv | 5 + dm/games/01007EF00011E000/hdr_tonemap.metal | 78 +++++++ dm/games/README.md | 1 + dm/games/game.py | 11 + dm/games/settings.tsv | 1 + dm/gfx/msl/hdr_merge.metal | 17 ++ dm/gfx/msl/present.metal | 12 +- dm/gfx/nvn/nvn_mtl.m | 225 ++++++++++++++++++-- dm/host/front_mac.m | 8 + dm/host/hle_hid.c | 3 + dm/host/tests/t_hid.c | 2 + dm/ios/Sources/front_ios.m | 17 ++ dm/tools/docs_patches.py | 2 + docs/agents/game-patches.md | 1 + docs/env_switches.tsv | 6 + docs/settings.md | 10 +- 17 files changed, 387 insertions(+), 20 deletions(-) create mode 100644 dm/games/01007EF00011E000/hdr_tonemap.metal create mode 100644 dm/gfx/msl/hdr_merge.metal diff --git a/dm/games/01007EF00011E000/README.md b/dm/games/01007EF00011E000/README.md index 1e08685b5..5eae792f0 100644 --- a/dm/games/01007EF00011E000/README.md +++ b/dm/games/01007EF00011E000/README.md @@ -70,6 +70,14 @@ Used at `DM_FPS_TARGET=0` and for the interval the framework keeps while the VFR - **Shrines at 20–27 fps (GitHub issues #534 and #598; current, the fix not yet seen in a shrine).** The reporters' shrines (Oman Au on a MacBook Air M2; every shrine on an M3) run at 20–27 fps where the open world holds 60. In the #598 log the slow stretches have about 1,000 draws a frame, as the fast ones, but a GPU time of about 36 ms against 9–13 ms. They have about 240 `passes ended to land them for a read` a frame, all `refused … other GPU writes`, and 280–330 `load-all store-all passes` against about 100. The probe on the Shrine of Resurrection save found the one write behind that refusal there: the GPU particle dispatch `9f8116baffddaf81` binds its three outputs with size 0, which counted as a write anywhere ([A storage buffer bound with size 0 counted as a write anywhere](../../../docs/reference/rendering.md#a-storage-buffer-bound-with-size-0-counted-as-a-write-anywhere)). With the fix, that save's world-load burst (1,200 refusals) is gone. Opening Oman Au's final gate ends the slowdown even out of view (#534): an emitter that stops with the gate would fit, but no shrine save was here to show it. - **Present snapshots (Mac).** The game's two swapchain textures held its CPU work to display ticks and the GPU at 444 MHz on an M2 Max ([present snapshots](../../../docs/reference/frame-rate.md#present-snapshots-the-games-swapchain-texture-free-at-once)). Judged by cost, the frame step stepped to 30 at a GPU time of 0.75 vsyncs. Zora's Domain from a Cemu save (120 Hz), frames on screen longer than 1/60 s by a quarter: 64 of 8,684 with neither, 41 with the GPU filler alone, 4 of 10,800 with snapshots alone (no step to 30 in 3 minutes, at 444 MHz throughout). +### HDR output (`DM_HDR`, current) + +The game renders its scene in RG11B10 float and tone maps it in one pass (`uking_pass_shader`, pixel shader `f7ee6fe10b925543`): x = min(scene, 10000) + glow, a mix of a luminance-preserving and a per-channel 1 − e^−x weighted by the luminance's own curve squared, then a saturation grade; the result goes into an RGB10A2 target, and the AA / upscale pass, the HUD and the copy into the RGBA8 swapchain's sRGB view all work on those linear values. With the profile's `hdr` key, `hdr_tonemap.metal` replaces that shader: it writes the game's result unchanged and, into the host's side texture, how much more the same pass gives with the curve's top raised to the screen's headroom. The raised curve is the game's up to 0.6 (`DM_HDR_KNEE`), then an exponential shoulder with the game's slope there that rises toward the peak instead of 1, so the picture below 0.6 and the colours stay the game's. The host adds the excess in the present where the picture still has the luminance the pass wrote ([`nvn_mtl.m`, HDR output](../../gfx/nvn/nvn_mtl.m)), so the HUD, menus, map and title, which have no tone mapping pass under them, stay the game's own. + +- **Measured on the Mac** (M1, macOS 27, headless, peak fixed at 4, screenshots as half floats with `DM_HDR_SHOT_RAW`): at peak 1 the picture's largest value is 1.0, the game's; at noon in a field about 4% of the pixels go above SDR white, the brightest sunlit grass and the clouds by the sun up to 2.4; at sunset 15%, the sky around the sun; the inventory 0.06% (the 3D Link behind it). A knee of 0.5 had lifted 13% of the field above white at noon. Through MetalFX at 2853x2007, the highlights stay on the sun, the sunlit flowers and Link's hair and none falls on the HUD. +- **On an iPhone Duo** (A20 Pro, iOS 27.1, MetalFX on): the headroom rose from 1.2 at launch to 8 once the layer showed extended values (screen brightness 50%); the shader compiled in 288 ms at load. Copying the picture into half float slots and scaling those made MetalFX run its HDR mode on half floats: 4.1 ms a present for 1915x1344 → 2853x2007, against about 2 ms in its perceptual mode on 8 bits, and heavier scenes dropped to 30 fps; the picture now stays 8-bit through MetalFX and the highlights are added after it. After about 8 minutes the phone reached thermal state serious (the frame step's 30 fps cap) at full screen brightness, by when iOS had lowered the headroom to 1. +- **Not done:** the profile's MSL is compiled at run time, not with the host's own libraries; a translucent HUD element over a highlight whose luminance stays within the tolerance can take some of its excess. + ## Rendering - **Resolved: the 3D scene black, only particles and the HUD drawn.** The engine binds almost every texture with `nvnCommandBufferBindTexturesDeferred` (`main+0xec2e64`, one handle, the pointer being the handle field inside the texture-sampler object): the GPU reads the handle when the command executes, and the game records its command blocks once and writes the handles afterwards. The host copied them at recording, mostly zeros, so the deferred-lighting composite read the dummy texture 0. The bind record keeps the source address and replay refreshes the handles first (`bh_fresh`). Found with a data-flow table from `DM_NVN_PASSDUMP_ALL`: the composite's `B` lines were all `tid=0 dummy` (without `_ALL` only the first draw of each pass is listed). diff --git a/dm/games/01007EF00011E000/game.tsv b/dm/games/01007EF00011E000/game.tsv index 5462e229b..978784d0b 100644 --- a/dm/games/01007EF00011E000/game.tsv +++ b/dm/games/01007EF00011E000/game.tsv @@ -18,6 +18,11 @@ app_category public.app-category.adventure-games pad_layout full # ---- Graphics ---- +# HDR output (nvn_mtl.m, HDR output; DM_HDR): the scene is RG11B10 float up to the tone mapping pass (uking_pass_shader, pixel shader f7ee6fe10b925543), which writes +# 1 - exp(-x) into an RGB10A2 target; the AA / upscale pass, the HUD and the copy into the RGBA8 swapchain's sRGB view all work on those linear values. hdr_tonemap.metal is +# that pass again: it writes the game's result unchanged and, beside it, how much brighter each pixel gets with the curve's top raised from 1 to the screen's headroom (the +# game's curve up to 0.6), which the host adds in the present where the 3D picture still shows, so the HUD and the menus stay the game's own +hdr f7ee6fe10b925543 hdr_tonemap.metal # The frame-interval controller (60 fps cuts in patches.py) and dynamic resolution read the GPU time between the frame's two timestamp reports: each report takes its segment's GPU start / end gpu_timestamps segment # 60 fps (30 on the Switch): the host decides each frame's step in whole vsyncs and holds the frame that long (rewriter stubs at setWaitVsyncInterval and VFR::updateInterval, diff --git a/dm/games/01007EF00011E000/hdr_tonemap.metal b/dm/games/01007EF00011E000/hdr_tonemap.metal new file mode 100644 index 000000000..5bcf953a0 --- /dev/null +++ b/dm/games/01007EF00011E000/hdr_tonemap.metal @@ -0,0 +1,78 @@ +// Breath of the Wild's tone mapping pass for HDR output (game.tsv hdr; nvn_mtl.m, HDR output): the host compiles this in place of the translated pixel shader +// f7ee6fe10b925543 (uking_pass_shader) and draws it with that shader's reflection, so the argument buffer below keeps the translation's bindings in their order. +// The game's pass, written out: +// x = min(scene, 10000) + glow (both RG11B10 float) +// l = dot(x, (0.2989, 0.587, 0.1145)), v(x) = 1 - exp(-x) (the luminance weights after the first come from the shader's constants, c1) +// t = saturate(mix(x * v(l) / l, v(x), v(l)^2)) (dark colours tone mapped by luminance, keeping their hue; bright ones per channel, going white) +// saturation grade: s = (r + g + b) * 2/3, g = c3[0].y * (1 - (s - 1)^2) + c3[0].x, out = m + (t - m) * g with m the largest channel +// written into an RGB10A2 target (which clamps it to [0, 1]); the AA pass, the HUD and the copy into the swapchain follow. This writes that result unchanged, and the same pass +// tone mapped up to the screen's headroom into the host's side texture, as what it adds to the SDR result (rgb) and the SDR result's luminance (a): nvn_mtl.m keeps it wherever +// the presented picture still has that luminance (hdr_merge) and adds it in the present. The HDR curve is the game's up to the knee (v = knee), then an exponential shoulder with the game's +// slope there (1 - knee) that rises toward the peak instead of 1 (with a peak of 1 it is the game's curve: nothing is added); the blend weight and the saturation grade are the +// game's own, so colours change only in how bright the highlights get. +#include +using namespace metal; + +struct Cb { float4 v[4096]; }; +struct Border { uint mask; float4 k[32]; }; // the translation's border colours: texture i sampled with its border where mask bit i is set (the chain's dm_border_t) +struct Args { + constant Cb *c1 [[id(0)]]; // the shader's constants: v[0] = (0.587, 0.1145, 2/3) + constant Cb *c3 [[id(1)]]; // uniform block 0: v[0].x / .y = the saturation grade's bias and scale (0.99, 0.18 in the field) + texture2d scene [[id(2)]]; + sampler scene_s [[id(3)]]; + texture2d glow [[id(4)]]; + sampler glow_s [[id(5)]]; + sampler scene_w [[id(6)]]; // the border-colour samplers and their mix, as the translation samples (its dm_bs) + sampler glow_w [[id(7)]]; + constant Border *border [[id(8)]]; +}; +struct In { float4 uv [[user(locn0)]]; }; +struct Out { float4 c [[color(0)]]; }; +// The host's HDR constants (nvn_mtl.m hdr_consts, fragment buffer 30): peak = the brightest value to tone map to, in SDR whites (the screen's EDR headroom; 1 = SDR); gain = the +// excess's scale; knee = the SDR value where the curve leaves the game's (DM_HDR_KNEE) +struct Hdr { float peak, gain, knee, pad; }; + +constant float kLog2e = 1.44269502f; + +static float3 game_curve(float3 x) { return 1.0f - exp2(x * -kLog2e); } +static float3 hdr_curve(float3 x, float peak, float knee) { + float xk = -log2(1.0f - knee) / kLog2e, slope = 1.0f - knee, room = peak - knee; // where the game's curve reaches the knee, and its slope there + float3 up = knee + room * (1.0f - exp2((x - xk) * (-slope * kLog2e / room))); + return select(game_curve(x), up, x > xk); +} +static float3 tex(texture2d t, sampler s, sampler w, constant Border *b, uint i, float2 uv) { + float3 r = t.sample(s, uv).xyz; + if (b->mask >> i & 1u) r += (t.sample(w, uv).xyz - r) * b->k[i].xyz; + return r; +} +static float3 grade(float3 t, float3 ref, float bias, float scale, float third2) { + float s = (ref.r + ref.g + ref.b) * third2 - 1.0f; + float g = scale * (1.0f - s * s) + bias; + float m = max(t.r, max(t.g, t.b)); + return m + (t - m) * g; +} + +fragment Out main0(In in [[stage_in]], float4 pos [[position]], constant Args &a [[buffer(0)]], constant Hdr &h [[buffer(30)]], + texture2d side [[texture(0)]]) { + float2 uv = select(in.uv.xy, float2(0.0f), isnan(in.uv.xy)); + float3 x = min(tex(a.scene, a.scene_s, a.scene_w, a.border, 0, uv), 10000.0f) + tex(a.glow, a.glow_s, a.glow_w, a.border, 1, uv); + float4 k = a.c1->v[0]; + float3 lw = float3(0.2989f, k.x, k.y); + float l = dot(x, lw); + float il = l > 0.0f ? 1.0f / l : 0.0f; + float vl = game_curve(float3(l)).x, w = vl * vl; + float3 sdr = saturate(mix(x * (vl * il), game_curve(x), w)); + float bias = a.c3->v[0].x, scale = a.c3->v[0].y; + float3 o = grade(sdr, sdr, bias, scale, k.z), shown = saturate(o); // what the game writes; what its target keeps + float3 more = 0.0f; + if (h.peak > 1.0001f) { + float knee = clamp(h.knee, 0.05f, 0.95f); + float3 hdr = clamp(mix(x * (hdr_curve(float3(l), h.peak, knee).x * il), hdr_curve(x, h.peak, knee), w), 0.0f, h.peak); + more = max(clamp(grade(hdr, sdr, bias, scale, k.z), 0.0f, h.peak) - shown, 0.0f) * max(h.gain, 0.0f); + } + uint2 p = uint2(pos.xy); + if (p.x < side.get_width() && p.y < side.get_height()) side.write(half4(half3(more), half(dot(shown, lw))), p); + Out out; + out.c = float4(o, 0.0f); + return out; +} diff --git a/dm/games/README.md b/dm/games/README.md index 7523e2699..d1d98c128 100644 --- a/dm/games/README.md +++ b/dm/games/README.md @@ -217,6 +217,7 @@ One row per profile; the notes are each profile's `README.md`. | `tex_dyn_slots` | `on` / `off` (the default): a shader whose buffer-texture handle the translator could not trace to one slot gets Ryujinx's fallback arrays (`__cb0_40000000`: 32 buffer textures and samplers indexed by the handle's texture / sampler ID; any other sampled texture indexes the whole pool without a key, [Texture handles from game memory index the pools](../../docs/reference/shaders.md#texture-handles-from-game-memory-index-the-pools)). With the key the replay puts slot indices in that shader's c2 texture and sampler tables (texture slot r → r, sampler slot r → r << 20) and binds slot k at element k, so the shader's own arithmetic picks the binding (`NVN_RB_TEXD` / `NVN_RB_SAMPD` in `nvn_mtl.m`). Right only where the handle comes from c2's tables: Octopath Traveler 0's 61 such vertex shaders read separate texture slots 0-10 and samplers 0-4 there and nothing else (FINAL FANTASY RESONANCE's 2,077 the texture table words 0x168-0x1c0 and sampler table words 0x568-0x590). Off, such shaders do not load. Not part of the shader chain version | | `gpu_timestamps` | What a timestamp report (`nvnCommandBufferReportCounter`) records: `submission` (the default) = the time the whole submission completed, the same for every report in it; `segment` = the GPU start (`TIMESTAMP_TOP`) or end (the other types) of the segment the report was encoded in (the host commits a submission in segments of 64 draws, `sub_cut` in `nvn_mtl.m`), so two reports of one submission bracket the GPU time between them, as on hardware. Breath of the Wild needs `segment`: its frame-interval controller and its dynamic resolution read the GPU time between the frame's first and last report, and with one time for both they read 0 (the game kept 60 fps in slow motion when the GPU could not keep up). `DM_NVN_GPU_TS=segment` / `submission` overrides it, also from `Documents/env.txt` | | `vconv_range` | `on` / `off` (the default): vertex attributes the replay converts on the CPU (integer shader inputs fed float-type formats and the like, `vconv_bind` in `nvn_mtl.m`) are converted only over the vertices each draw reads (first / count, or the smallest and largest index of a direct indexed draw, kept while the index pages stay unwritten) instead of the whole bound range. Age of Calamity binds whole mesh pools: in a battle that was millions of vertices a frame. Indirect draws, per-instance streams and indices with a GPU write queued on them keep the whole range. `DM_NVN_VCONV_CHECK=1` scans the indices again at every reuse and reports differences; `DM_NVN_VCONV_RANGE=0` / `1` overrides it | +| `hdr` | ` ` (none by default): HDR output. The file, an MSL source in the profile folder, replaces the game's tone mapping pixel shader of that key and keeps its bindings (the host draws it with the translated shader's reflection): it writes the game's own result as before and, into the host's side texture (fragment texture 0, `RGBA16Float`), how much brighter each pixel gets tone mapped up to the peak the host passes at fragment buffer 30 (`hdr_consts`), and the SDR luminance it wrote. The host adds that excess in the present where the picture still has that luminance, into an extended-range layer, and turns present snapshots on (`nvn_mtl.m`, HDR output; `DM_HDR`). The file goes into the host as a string (`DM_GAME_HDR_MSL`) and is compiled when the game loads that shader. Breath of the Wild: `f7ee6fe10b925543 hdr_tonemap.metal` | | `shader_key_skip` | `+` (none by default): control-section bytes every shader key hashes as zeros, in `shader_split.py` (`key_ctl`) and the host (`hle_nvn.c` `keys_hash`), for an engine that shares one control section among shaders that differ only there. Batman: Arkham Knight: `0x770+0x78`, GLSLC's compile information (+0x778..+0x791 and +0x7d0..+0x7e7); without it the game asks for shaders the map does not have, each the control section of one record with the code area of another. It must stay clear of `+0x700..+0x75f`, which the tools read; it is a shader chain input, as every `shader_*` key | | `shader_groups`, `shader_note` | `shader_split.py` source groups and the map header note (Engage: `engine` for `data.unity3d` and `Resources/`, `assets` for the Addressables bundles). Both are shader inputs, the note too: changing either one, even by a word, builds every shader of the game again. Keep them as they are when a provisional profile moves into `dm/games` | diff --git a/dm/games/game.py b/dm/games/game.py index 6089fe9f0..69fd5df3e 100644 --- a/dm/games/game.py +++ b/dm/games/game.py @@ -148,6 +148,14 @@ def header(g): if dmo not in ('gl', 'zero'): sys.exit(f'{g.id}: depth_mode must be gl or zero: {dmo}') api = g.get('device_api', '55.15') # the NVN API major.minor version nvnDeviceGetInteger reports (hle_nvn.c) if not re.fullmatch(r'\d+\.\d+', api): sys.exit(f'{g.id}: device_api must be major.minor: {api}') + # HDR output (nvn_mtl.m, HDR output): the key of the game's tone mapping pixel shader and the profile's MSL file that replaces it, + # with the same bindings, tone mapping up to the screen's headroom; the file goes into the host as a string (DM_GAME_HDR_MSL), compiled when the host loads that shader + hdr = g.words('hdr', '') + if hdr and (len(hdr) != 2 or not re.fullmatch(r'[0-9a-f]{16}', hdr[0]) or not (g.dir / hdr[1]).is_file()): + sys.exit(f'{g.id}: hdr is : {" ".join(hdr)}') + hdr_msl = (g.dir / hdr[1]).read_text() if hdr else '' + if not hdr_msl.isascii(): sys.exit(f'{g.id}: {hdr[1]} must be ASCII (it goes into a C string)') + cs = lambda s: '"' + s.replace('\\', '\\\\').replace('"', '\\"').replace('\n', '\\n"\\\n"') + '"' # a C string over several lines src = f'dm/games/{g.id}' if g.dir.parent == HERE else str(g.dir) L = [f'// Generated by dm/games/game.py from {src}/game.tsv; do not edit', '#pragma once', @@ -189,6 +197,9 @@ def header(g): f'#define DM_GAME_TEX_DYN_SLOTS {int(tds == "on")}', f'#define DM_GAME_GPU_TS_SEGMENT {int(gts == "segment")}', f'#define DM_GAME_VCONV_RANGE {int(vcr == "on")}', + f'#define DM_GAME_HDR {int(bool(hdr))}', + f'#define DM_GAME_HDR_KEY {q(hdr[0] if hdr else "")}', + f'#define DM_GAME_HDR_MSL {cs(hdr_msl)}', *([f'#define DM_GAME_NVN_API_MAJOR {api.split(".")[0]}', f'#define DM_GAME_NVN_API_MINOR {api.split(".")[1]}'] if g.get('device_api', '') else []), # without the key: by the SDK version (hle_nvn.c nvn_api) ''] return '\n'.join(L) diff --git a/dm/games/settings.tsv b/dm/games/settings.tsv index 00e1ddd10..ca7c6545b 100644 --- a/dm/games/settings.tsv +++ b/dm/games/settings.tsv @@ -22,6 +22,7 @@ DM_PRESENT_UPSCALE choice off=Standard,metalfx=MetalFX off always Game Upscaling DM_LANG choice languages build:DM_LANG languages>1 Game Game language level=basic DM_FULLSCREEN choice 0=Window,1=Fullscreen 0 platform:mac Game Display mode level=basic DM_SHOW_FPS toggle - build:DM_SHOW_FPS platform:ios Game Frame-rate counter +DM_HDR toggle - 1 DM_GAME_HDR!=0 Game HDR A little more GPU time and memory; bright scenes use more power. Needs a screen with HDR. DM_TOUCH_HAPTICS toggle - 1 platform:ios Controls Touch feedback DM_RUMBLE toggle - 1 always Controls Game rumble level=basic DM_RUMBLE_GAIN range 0:3:0.1 1.0 always Controls Rumble strength needs=DM_RUMBLE=1 diff --git a/dm/gfx/msl/hdr_merge.metal b/dm/gfx/msl/hdr_merge.metal new file mode 100644 index 000000000..0dbbf88e0 --- /dev/null +++ b/dm/gfx/msl/hdr_merge.metal @@ -0,0 +1,17 @@ +// HDR output's present copy (nvn_mtl.m hdr_merge): the swapchain's bytes into the ring slot as they are (pic), and the side texture's excess into the slot's highlights texture (hx) +// where the picture's luminance, decoded from sRGB, is still what the tone mapping wrote (within 1.2 % + 5 %, fading out over as much again). scale: the tone mapping target's +// share of the side texture +#include +using namespace metal; +struct V { float4 p [[position]]; float2 uv; }; +vertex V vs(uint id [[vertex_id]]) { float2 uv = float2((id << 1) & 2, id & 2); V o; o.p = float4(uv * float2(2, -2) + float2(-1, 1), 0, 1); o.uv = uv; return o; } +struct M { float2 scale; float2 pad; }; +struct O { float4 pic [[color(0)]]; float4 hx [[color(1)]]; }; +fragment O fs(V in [[stage_in]], texture2d pic [[texture(0)]], texture2d side [[texture(1)]], sampler s [[sampler(0)]], constant M &m [[buffer(0)]]) { + float4 v = pic.read(uint2(in.p.xy)); + float3 c = select(pow((v.rgb + 0.055) / 1.055, float3(2.4)), v.rgb / 12.92, v.rgb <= 0.04045); + float4 g = side.sample(s, in.uv * m.scale); + float tol = 0.012 + 0.05 * g.a; + O o; o.pic = v; o.hx = float4(g.rgb * (1 - smoothstep(tol, 2 * tol, abs(dot(c, float3(0.2989, 0.587, 0.1145)) - g.a))), 0); + return o; +} diff --git a/dm/gfx/msl/present.metal b/dm/gfx/msl/present.metal index 5bfdfeac4..379fe3029 100644 --- a/dm/gfx/msl/present.metal +++ b/dm/gfx/msl/present.metal @@ -1,7 +1,13 @@ -// The present: the frame drawn into the drawable, cropped (nvn_mtl.m present_pipeline) +// The present: the frame drawn into the drawable, cropped (nvn_mtl.m present_pipeline). hdr.x (present_mode): 1 = an 8-bit picture, sRGB-encoded bytes as the Switch scans them +// out, into an extended-range (linear) target, else 0; hdr.y: add texture 1's highlights (HDR output, nvn_mtl.m snap_excess), read at in.uv * ex.zw + ex.xy #include using namespace metal; struct V { float4 p [[position]]; float2 uv; }; vertex V vs(uint id [[vertex_id]]) { float2 uv = float2((id << 1) & 2, id & 2); V o; o.p = float4(uv * float2(2, -2) + float2(-1, 1), 0, 1); o.uv = uv; return o; } -struct Crop { float4 rect; float4 edge; }; -fragment float4 fs(V in [[stage_in]], texture2d t [[texture(0)]], sampler s [[sampler(0)]], constant Crop& c [[buffer(0)]]) { return t.sample(s, clamp(in.uv * c.rect.zw + c.rect.xy, c.edge.xy, c.edge.zw)); } +struct Crop { float4 rect; float4 edge; float4 hdr; float4 ex; }; +fragment float4 fs(V in [[stage_in]], texture2d t [[texture(0)]], texture2d x [[texture(1)]], sampler s [[sampler(0)]], constant Crop& c [[buffer(0)]]) { + float4 v = t.sample(s, clamp(in.uv * c.rect.zw + c.rect.xy, c.edge.xy, c.edge.zw)); + if (c.hdr.x == 1) v.rgb = select(pow((v.rgb + 0.055) / 1.055, float3(2.4)), v.rgb / 12.92, v.rgb <= 0.04045); + if (c.hdr.y > 0) v.rgb += x.sample(s, in.uv * c.ex.zw + c.ex.xy).rgb; + return v; +} diff --git a/dm/gfx/nvn/nvn_mtl.m b/dm/gfx/nvn/nvn_mtl.m index 53b6abd3f..62bd2ea3e 100644 --- a/dm/gfx/nvn/nvn_mtl.m +++ b/dm/gfx/nvn/nvn_mtl.m @@ -387,6 +387,13 @@ static void udc_selftest(void) { MLOG("UDC self-check %s %s %lux%lu first value %08x, values differing from it %lu / %lu", g_dev.name.UTF8String, depth ? "Depth32F_S8" : "RGBA8", (unsigned long)w, (unsigned long)h, px[0], (unsigned long)bad, (unsigned long)(w * h)); } } +// HDR output (see hdr_lib): the replaced tone mapping shader's constants (the profile's MSL, struct Hdr): peak = the brightest value it tone maps to, in SDR whites; gain = the +// excess's scale; knee = the SDR value from which its curve rises toward the peak instead of 1; and the side texture it writes +typedef struct { float peak, gain, knee, pad; } hdr_consts; +static id g_hdr_buf; static id g_hdr_side; +#define HDR_SIDE_W 1920 // the side texture's size: the largest tone mapping target (Breath of the Wild: 1600x1120 at render scale 1 docked in the Duo's shape); the shader writes no farther +#define HDR_SIDE_H 1440 +int mtl_hdr_on(void); void mtl_init(void) { if (g_dev) return; const char *vbc = getenv("DM_NVN_VBUF_CACHE"); g_vbuf_cache = !(vbc && !strcmp(vbc, "0")); @@ -455,14 +462,23 @@ void mtl_init(void) { g_vt = rec_table([g_dev newArgumentTableWithDescriptor:atd error:&ate]); if (!g_vt) MDIE("failed to create the vertex argument table: %s", ate.localizedDescription.UTF8String); g_vt_valid = 0; g_ft = rec_table([g_dev newArgumentTableWithDescriptor:atd error:&ate]); if (!g_ft) MDIE("failed to create the fragment argument table: %s", ate.localizedDescription.UTF8String); + if (mtl_hdr_on()) { // bound once, at fragment buffer slot 30 and texture slot 0: the table keeps its bindings, and draws set only the slots their shaders use + g_hdr_buf = [g_dev newBufferWithLength:sizeof(hdr_consts) options:MTLResourceStorageModeShared]; + MTLTextureDescriptor *sd = [MTLTextureDescriptor texture2DDescriptorWithPixelFormat:MTLPixelFormatRGBA16Float width:HDR_SIDE_W height:HDR_SIDE_H mipmapped:NO]; + sd.storageMode = MTLStorageModePrivate; sd.usage = MTLTextureUsageShaderRead | MTLTextureUsageShaderWrite; + g_hdr_side = [g_dev newTextureWithDescriptor:sd]; + if (!g_hdr_buf || !g_hdr_side) MDIE("failed to create the HDR constants / side texture"); + *(hdr_consts *)g_hdr_buf.contents = (hdr_consts){1, 1, 0.6f, 0}; g_hdr_side.label = @"HDR excess"; + res_add(g_hdr_buf); res_add(g_hdr_side); [g_ft setAddress:g_hdr_buf.gpuAddress atIndex:30]; [g_ft setTexture:g_hdr_side.gpuResourceID atIndex:0]; + } g_ct = rec_table([g_dev newArgumentTableWithDescriptor:atd error:&ate]); if (!g_ct) MDIE("failed to create the compute argument table: %s", ate.localizedDescription.UTF8String); g_up_ct = [g_dev newArgumentTableWithDescriptor:atd error:&ate]; if (!g_up_ct) MDIE("failed to create the upload compute argument table: %s", ate.localizedDescription.UTF8String); MTL4ArgumentTableDescriptor *pad = [MTL4ArgumentTableDescriptor new]; - pad.maxTextureBindCount = 1; pad.maxSamplerStateBindCount = 1; pad.maxBufferBindCount = 1; + pad.maxTextureBindCount = 2; pad.maxSamplerStateBindCount = 1; pad.maxBufferBindCount = 1; // texture 1: HDR output's highlights (snap_excess) g_present_args = [g_dev newArgumentTableWithDescriptor:pad error:&ate]; if (!g_present_args) MDIE("failed to create the present argument table: %s", ate.localizedDescription.UTF8String); for (int i = 0; i < 3; i++) { - g_present_uv[i] = [g_dev newBufferWithLength:sizeof(float) * 8 options:MTLResourceStorageModeShared]; + g_present_uv[i] = [g_dev newBufferWithLength:sizeof(float) * 16 options:MTLResourceStorageModeShared]; if (!g_present_uv[i]) MDIE("failed to create present UV buffer[%d]", i); res_add(g_present_uv[i]); } @@ -1689,7 +1705,7 @@ static void mtl_finish_gpu_at(int site) { // ================= Render replay ================= #include "../../host/front.h" -static int g_red_once[142]; +static int g_red_once[143]; #define RED_ONCE(k, ...) do { if (!g_red_once[k]++) MLOG("[RED] " __VA_ARGS__); } while (0) // NVN vertex streams 0–15 → Metal vertex buffer slots: 0–13 go to 16–29 (30 is the constant fallback buffer), 14 and 15 to 14 and 15. Metal has only the 31 slots 0–30; translated shaders use only buffer(0) (the argument buffer), // and the host's own vertex functions read no vertex buffers. 13 Sentinels uses stream 15, and the old 16 + stream went out of range (Metal assertion: buffer index (31) must be < 31) @@ -1941,6 +1957,66 @@ static void pkg_absorb_take(NvPkg *pk, const tex_t *t) { } free(e.edge[0]); free(e.edge[1]); } +// ---------- HDR output (profile key hdr, DM_HDR) ---------- +// The game keeps drawing its SDR picture as on the Switch. A profile that names its game's tone mapping pixel shader (game.tsv hdr) has that shader replaced by its own MSL +// (DM_GAME_HDR_MSL, compiled when the shader loads: hdr_lib), which writes the game's own result as before and, beside it, into the host's side texture (g_hdr_side: fragment +// texture slot 0 of the draw table, RGBA16Float, indexed by the tone mapping target's pixels) how much brighter each pixel gets tone mapped up to the screen's headroom (rgb) and +// the SDR luminance it wrote (a). Each present is copied on the game's queue right after the frame's work (the present snapshots, which HDR output turns on: snap_take); the copy +// is a draw that keeps the game's 8-bit picture as it is and writes beside it the side texture's excess where the picture still has the luminance the tone mapping wrote, that is +// where neither the HUD nor a fade nor a later effect has drawn over the 3D picture (hdr_merge, into the slot's RG11B10Float highlights texture). The present scales the 8-bit +// picture as an SDR present does (MetalFX in its perceptual mode, or the bilinear pass), decodes it to linear and adds the highlights, into an extended-range layer (the front +// ends: RGBA16Float, extended linear sRGB; 1.0 = SDR white). A frame without the tone mapping pass (menus, the map), or with the peak at 1, is copied without highlights: the HUD, +// the menus and the title stay the game's own. The replaced shader reads its peak from the host's constants at fragment buffer 30 (hdr_consts). +// (Copying into half float slots and scaling those instead made MetalFX run in its HDR mode on half floats: 4.1 ms a present on an iPhone Duo, 1915x1344 → 2853x2007, against +// about 2 ms in the perceptual mode on 8 bits, which pushed heavier scenes off 60 fps.) +// DM_HDR=0 turns it off (the iOS front end sets it when the screen has no EDR headroom); DM_HDR_PEAK= fixes the peak in SDR whites (default: the headroom the front end reports, +// mtl_hdr_headroom, at most DM_HDR_MAX, default 8; DM_CTL hdrpeak sets it while the game runs); DM_HDR_GAIN= scales the excess (default 1); DM_HDR_KNEE= is the SDR value +// where the curve leaves the game's (default 0.6: Breath of the Wild on the Mac, peak 4: at noon about 4% of the picture above SDR white, the brightest sunlit grass and the +// clouds by the sun, up to 2.4; at sunset 15%, the sky round the sun; 0.5 had lifted most of the sunlit grass above white at noon, 13%; DM_CTL hdrknee) +static int g_hdr = -1; +int mtl_hdr_on(void) { + if (g_hdr < 0) { + const char *e = getenv("DM_HDR"), *ps = getenv("DM_PRESENT_SNAP"); + int on = DM_GAME_HDR && !(e && *e && !atoi(e)), nosnap = on && ps && *ps && atoi(ps) <= 0; // the highlights travel with the present snapshots' copy (snap_take) + if (nosnap) on = 0; + if (DM_GAME_HDR) MLOG("HDR output (DM_HDR): %s", on ? "on: pixel shader " DM_GAME_HDR_KEY " also writes its highlights above SDR white, added in the present where the 3D picture shows" : + nosnap ? "off: present snapshots are off (DM_PRESENT_SNAP=0), and their copy carries the highlights" : "off"); + g_hdr = on; + } + return g_hdr; +} +static int g_hdr_drawn, g_hdr_rw, g_hdr_rh; // the tone mapping pass drew since the last present copy, into a target of this size (draw and snap_take, both on the submitting thread) +static void hdr_drawn(const tex_t *t) { g_hdr_drawn = 1; g_hdr_rw = (int)MIN(t->d.width, HDR_SIDE_W); g_hdr_rh = (int)MIN(t->d.height, HDR_SIDE_H); } +static _Atomic uint32_t g_hdr_head = 0x3f800000; // the screen's headroom (float bits) as the front end last reported it; 1 = SDR +void mtl_hdr_headroom(float h) { if (h >= 1 && h < 1000) { uint32_t b; memcpy(&b, &h, sizeof b); atomic_store_explicit(&g_hdr_head, b, memory_order_relaxed); } } +float mtl_hdr_peak(void) { + if (!g_hdr_buf) return 1; + return ((volatile hdr_consts *)g_hdr_buf.contents)->peak; +} +static _Atomic uint32_t g_hdr_set, g_hdr_set_knee; // a peak / knee set while the game runs (float bits; DM_CTL hdrpeak / hdrknee, hle_hid.c), 0 = none +void mtl_hdr_set_peak(float p) { uint32_t b = 0; if (p > 0) { p = fmaxf(1, p); memcpy(&b, &p, sizeof b); } else b = 0x7fc00000; atomic_store(&g_hdr_set, b); } +void mtl_hdr_set_knee(float k) { uint32_t b; k = fminf(fmaxf(k, 0.05f), 0.95f); memcpy(&b, &k, sizeof b); atomic_store(&g_hdr_set_knee, b); } +// Once a present: the peak follows the headroom, eased over about half a second so that a change of the screen's brightness does not flash the highlights +static void hdr_update(void) { + if (!g_hdr_buf) return; + static float fixed = -1, top = 8, gain = 1, knee = 0.6f, peak = 1; + if (fixed < 0) { + const char *p = getenv("DM_HDR_PEAK"), *m = getenv("DM_HDR_MAX"), *g = getenv("DM_HDR_GAIN"), *k = getenv("DM_HDR_KNEE"); fixed = 0; + if (p && *p) peak = fixed = fmaxf(1, (float)atof(p)); + if (m && *m) top = fmaxf(1, (float)atof(m)); + if (g && *g) gain = fmaxf(0, (float)atof(g)); + if (k && *k) knee = fminf(fmaxf((float)atof(k), 0.05f), 0.95f); + MLOG("HDR: peak %s, at most %.2f; excess x%.2f; the curve leaves the game's at %.2f", fixed ? [NSString stringWithFormat:@"%.2f (DM_HDR_PEAK)", fixed].UTF8String : "the screen's headroom", top, gain, knee); + } + uint32_t set = atomic_exchange(&g_hdr_set, 0), setk = atomic_exchange(&g_hdr_set_knee, 0); + if (set) { float v; memcpy(&v, &set, sizeof v); fixed = v == v ? v : 0; if (fixed) peak = fixed; } // NaN: back to the headroom + if (setk) memcpy(&knee, &setk, sizeof knee); + uint32_t b = atomic_load_explicit(&g_hdr_head, memory_order_relaxed); float h; memcpy(&h, &b, sizeof h); + float want = fixed ? fixed : fminf(h, top); + peak += (want - peak) * 0.06f; if (fabsf(want - peak) < 0.002f) peak = want; + hdr_consts *c = g_hdr_buf.contents; c->peak = peak; c->gain = gain; c->knee = knee; + static float logged = -1; if (fabsf(peak - logged) > 0.25f || (peak == want && logged != want && fabsf(peak - logged) > 0.01f)) { logged = peak; MLOG("HDR: peak %.2f (screen headroom %.2f)", peak, h); } +} // Usage an NVN image takes: shader writes, and for R32Uint / R32Sint the shader-atomic usage too, since Metal runs texture atomics only on such textures (dummy_img_atomic) and NVN does // not say which images a shader will use atomically static inline int pf_r32i(MTLPixelFormat pf) { return pf == MTLPixelFormatR32Uint || pf == MTLPixelFormatR32Sint; } @@ -3829,6 +3905,7 @@ @interface NvShader : NSObject { @public id fn, tf_fn, discard_fn, id ab_buf; uint64_t ab_gpu; size_t ab_off; uint64_t ab_pass; uint32_t ab_binst; uint64_t ab_pgen; // ab_pass: serial of the render pass that encoded this argument block - reused only within the same pass (the arena holding a block from another pass / another submission may already have been reclaimed and rewritten); ab_pgen: g_psnap_gen when it was encoded (pools, pool_gpu) id bflib; int bflib_tried; NSMutableDictionary> *bffn; // bflib: readable-tile version _bf (merge B step 2) uint64_t mis[4]; // mis: texture slots 0-255 already reported for a type mismatch (tex_mis_note) + uint8_t hdr; // the profile's HDR tone mapping shader (hdr_lib): its draws write the side texture (hdr_drawn) NSString *ventry; id vfn; int vfn_state; id pfn; int pfn_state, has_pull; id afn[4]; int afn_state, has_alpha; @@ -3978,6 +4055,18 @@ void mtl_set_depth_zero(int on) { os_unfair_lock_lock(&g_fn4_lock); MTL4FunctionDescriptor *d = [g_fn4 objectForKey:f]; os_unfair_lock_unlock(&g_fn4_lock); return d; } +// HDR output (see hdr_drawn): the profile's tone mapping shader in place of the library of its key; its reflection is the translated shader's. nil for any other key, or when it does +// not compile (a red line; the game's shader is drawn and the presents carry no excess). Compiled from source once, when the game loads the shader (0.3 ms on an M1 with the Metal +// cache warm, 288 ms on an iPhone Duo the first time) +static id hdr_lib(NSString *pc) { + if (!DM_GAME_HDR || !mtl_hdr_on() || ![pc isEqualToString:@DM_GAME_HDR_KEY "_b"]) return nil; + NSError *e = nil; uint64_t t0 = clock_gettime_nsec_np(CLOCK_MONOTONIC_RAW); + id lib = [g_dev newLibraryWithSource:@(DM_GAME_HDR_MSL) options:nil error:&e]; + if (!lib) { MLOG("[RED] HDR: the profile's tone mapping shader does not compile, drawing the game's %s: %s", pc.UTF8String, e.localizedDescription.UTF8String); return nil; } + t_lib_layer = 0; + MLOG("HDR: tone mapping shader %s from the profile (compiled in %.1f ms)", pc.UTF8String, (clock_gettime_nsec_np(CLOCK_MONOTONIC_RAW) - t0) / 1e6); + return lib; +} // A function that declares function constants, specialized with cv (an empty set keeps every default). On the Metal 4 path the pipeline's descriptor specializes it (fn4_note // records cv, and MTL4Compiler compiles it with the pipeline), so the MTLFunction stays unspecialized: its reflection and argument encoders are the same. MTLLibrary's // newFunctionWithName:constantValues: compiles through the MTLDevice, synchronously, on the thread that loads the shader: when Animal Crossing loaded such a shader during play, @@ -4032,7 +4121,8 @@ @implementation FnDev @end } if (stage == 2 && !j) return nil; // missing geometry translation: the caller's red line says so NSString *lp = [NSString stringWithFormat:@"%@/%@.metallib", g_msl_dir, pc]; - id lib = lib_from_file(lp, &e); + id lib = hdr_lib(pc); int hdr = lib != nil; + if (!lib) lib = lib_from_file(lp, &e); if (!lib || !j) { MLOG("[RED] failed to load shader %s: %s", pc.UTF8String, e ? e.localizedDescription.UTF8String : "no reflection json"); return nil; } if (![j[@"chain"] isEqual:@DMSH_CHAIN]) { MLOG("[RED] shader %s has translation chain version %s, but this host expects %s", pc.UTF8String, [[j[@"chain"] description] UTF8String], DMSH_CHAIN); return nil; } if (layer && !j[@"entry_layer"]) { MLOG("[RED] vertex shader %s has no layer entry points", pc.UTF8String); return nil; } @@ -4065,7 +4155,7 @@ @implementation FnDev @end for (NSUInteger i = 0; i < reads.count; i++) { int k = [reads[i] intValue]; if (k >= 0 && k < GEO_CAPTURE_WORDS) o->geo_slot[k] = (int16_t)i; } o->geo_words = (uint32_t)reads.count; } } - o->fn = fn4_note([lib newFunctionWithName:entry], lib, entry, nil); o->stage = stage; strlcpy(o->key, pc.UTF8String, sizeof o->key); o->ventry = entry; + o->fn = fn4_note([lib newFunctionWithName:entry], lib, entry, nil); o->stage = stage; strlcpy(o->key, pc.UTF8String, sizeof o->key); o->ventry = entry; o->hdr = (uint8_t)hdr; // writes the layer (render_target_array_index, named in the library's metadata): its passes are layered and its pipelines get an input class (pso_desc). A library with layer entry points // names it for those alone: its main0 / main0_z write no layer if (stage == 0) o->layer_out = layer || (t_lib_layer && !j[@"entry_layer"]); @@ -10004,6 +10094,7 @@ static void draw_(const p_draw *dw, const p_draw_idx *di) { g_kd_force = kd_draw(pr, dw, di); if (pr->sh[0]) encode_args(en, pr->sh[0], 0, 1); if (pr->sh[1] && !S.rast_discard) encode_args(en, pr->sh[1], 1, 0); + if (pr->sh[1] && pr->sh[1]->hdr && !S.rast_discard && R.rt.n && R.rt.color[0]) hdr_drawn(R.rt.color[0]); // HDR output: this frame's side texture is written // Binding a texture view can upgrade its texture first (tex_upgrade: a format view of a texture made without PixelFormatView), and the upgrade's copy goes through blit(), // which ends the render pass: en is then an ended encoder, and drawing on it crashed in the driver (AGX reads its released context: Need for Speed: Hot Pursuit Remastered, // a 4096x1024 texture sampled through another format's view at its first loading screen). The texture is upgraded now, so the draw is encoded again in a new pass @@ -11230,6 +11321,8 @@ void mtl_res_counts(uint32_t tick, const char *why) { static struct { uint64_t frame; char path[1024]; } g_shotq[SHOTQ_MAX]; static int g_nshotq; static pthread_mutex_t g_shotq_mu = PTHREAD_MUTEX_INITIALIZER; static int present_encode(id cb, id src, id dst, const int crop[4]); +static int present_crop_rect(id src, const int crop[4], NSUInteger r[4]); +static id snap_excess(id src, MTLPixelFormat dst); enum { PRESENT_FAILED, PRESENT_STANDARD, PRESENT_METALFX }; static int present_offscreen_m4(id src, const int crop[4], id out, void (^done)(int)); static int present_offscreen_wait(id src, const int crop[4], id out); @@ -11281,6 +11374,25 @@ static void shot(id src, const char *path, int screen, const int cro if (e && sscanf(e, "%d%c%d%c", &w, &sep, &h, &extra) == 3 && sep == 'x' && w > 0 && h > 0) { out_w = w; out_h = h; } } id read = src; + const char *hdr_raw = getenv("DM_HDR_SHOT_RAW"); + if (screen && g_q4 && hdr_raw && atoi(hdr_raw) && snap_excess(src, MTLPixelFormatRGBA16Float)) { + // HDR output: the PNG is the game's SDR picture; DM_HDR_SHOT_RAW=1 also writes the picture as the present draws it into an extended-range layer, highlights included, at + // DM_NVN_SHOT_OUT's size (through MetalFX with DM_PRESENT_UPSCALE=metalfx) or else at the size of its valid rectangle (path.rgba16f: a line "RGBA16F ", + // then the rows, linear, 1 = SDR white) + NSUInteger r[4]; if (!present_crop_rect(src, crop, r)) return; + size_t rw = out_w ? (size_t)out_w : r[2], rh = out_w ? (size_t)out_h : r[3], rrow = rw * 8; + MTLTextureDescriptor *d = [MTLTextureDescriptor texture2DDescriptorWithPixelFormat:MTLPixelFormatRGBA16Float width:rw height:rh mipmapped:NO]; + d.usage = MTLTextureUsageRenderTarget | MTLTextureUsageShaderRead; d.storageMode = MTLStorageModePrivate; + id out = [g_dev newTextureWithDescriptor:d]; + if (out && present_offscreen_wait(src, crop, out)) { + id fb = [g_dev newBufferWithLength:rrow * rh options:MTLResourceStorageModeShared]; + id cb = q_cb(); id b = [cb blitCommandEncoder]; + [b copyFromTexture:out sourceSlice:0 sourceLevel:0 sourceOrigin:MTLOriginMake(0, 0, 0) sourceSize:MTLSizeMake(rw, rh, 1) toBuffer:fb destinationOffset:0 destinationBytesPerRow:rrow destinationBytesPerImage:rrow * rh]; + [b endEncoding]; [cb commit]; [cb waitUntilCompleted]; + char fp[1100]; snprintf(fp, sizeof fp, "%s.rgba16f", path); FILE *f = fopen(fp, "wb"); + if (f) { fprintf(f, "RGBA16F %zu %zu\n", rw, rh); fwrite(fb.contents, 1, rrow * rh, f); fclose(f); MLOG("screenshot half floats %zux%zu → %s", rw, rh, fp); } + } else MLOG("[RED] cannot draw the HDR screenshot's half floats %zux%zu", rw, rh); + } if (out_w && screen) { MTLTextureDescriptor *d = [MTLTextureDescriptor texture2DDescriptorWithPixelFormat:MTLPixelFormatBGRA8Unorm width:out_w height:out_h mipmapped:NO]; d.usage = MTLTextureUsageRenderTarget | MTLTextureUsageShaderRead; d.storageMode = MTLStorageModePrivate; @@ -11579,6 +11691,11 @@ static int present_crop_rect(id src, const int crop[4], NSUInteger r } r[0] = crop[0]; r[1] = crop[1]; r[2] = crop[2]; r[3] = crop[3]; return 1; } +// How the present pass turns the picture's values into the target's (the shader's hdr.x): 0 as they are; 1 an 8-bit picture, sRGB-encoded bytes as the Switch scans them out, +// into an extended-range (linear) target (HDR output) +static float present_mode(id src, MTLPixelFormat dst) { + return dst == MTLPixelFormatRGBA16Float && src.pixelFormat != MTLPixelFormatRGBA16Float; +} static int present_encode(id cb, id src, id dst, const int crop[4]) { NSUInteger r[4]; if (!present_crop_rect(src, crop, r)) return 0; NSUInteger cx = r[0], cy = r[1], cw = r[2], ch = r[3]; @@ -11600,8 +11717,9 @@ static int present_encode(id cb, id src, id en = [cb renderCommandEncoderWithDescriptor:rp]; [en setViewport:vp]; [en setRenderPipelineState:g_blit_pso]; [en setFragmentTexture:present_raw_src(src) atIndex:0]; [en setFragmentSamplerState:g_blit_samp atIndex:0]; - float uv[8] = {(float)cx / src.width, (float)cy / src.height, (float)cw / src.width, (float)ch / src.height, - (cx + 0.5f) / src.width, (cy + 0.5f) / src.height, (cx + cw - 0.5f) / src.width, (cy + ch - 0.5f) / src.height}; + [en setFragmentTexture:present_raw_src(src) atIndex:1]; // the highlights' slot, unread here (hdr.y = 0) + float uv[16] = {(float)cx / src.width, (float)cy / src.height, (float)cw / src.width, (float)ch / src.height, + (cx + 0.5f) / src.width, (cy + 0.5f) / src.height, (cx + cw - 0.5f) / src.width, (cy + ch - 0.5f) / src.height, present_mode(src, dst.pixelFormat)}; if (atomic_load_explicit(&g_origin_ll, memory_order_relaxed)) { uv[1] += uv[3]; uv[3] = -uv[3]; } // lower-left origin: the window image is stored bottom-up [en setFragmentBytes:uv length:sizeof uv atIndex:0]; [en drawPrimitives:MTLPrimitiveTypeTriangle vertexStart:0 vertexCount:3]; @@ -11613,7 +11731,7 @@ static int present_encode(id cb, id src, id sc; id in, out; uint64_t made_at; int made; NSUInteger refused[6]; } fx_state; +typedef struct { id sc; id in, out; uint64_t made_at; int made; NSUInteger refused[6], from[2]; } fx_state; // from: the scaled rectangle's origin static fx_state g_fx_present; // only the present thread encodes or replaces the scaler static id g_fx_comp; static int g_fx_mode = -1; // 1 = metalfx, 0 = off @@ -11636,6 +11754,7 @@ static int fx_wanted(MTLViewport vp, NSUInteger cw, NSUInteger ch) { // remake_wait: fx is the present thread's, whose textures presents still on the GPU may read: a new scaler waits for them and for FX_REMAKE_PRESENTS presents since the last static id fx_encode(fx_state *fx, id cb, id raw, const NSUInteger r[4], MTLViewport vp, MTLPixelFormat out_pf, int remake_wait) { NSUInteger cw = r[2], ch = r[3], ow = (NSUInteger)vp.width, oh = (NSUInteger)vp.height; + if (out_pf == MTLPixelFormatRGBA16Float && raw.pixelFormat != MTLPixelFormatRGBA16Float) out_pf = raw.pixelFormat; // HDR output's extended-range target: 8 bits scaled into 8, decoded by the pass if (fx_mode() != 1 || !fx_wanted(vp, cw, ch)) return nil; if (!fx->sc || fx->in.width != cw || fx->in.height != ch || fx->in.pixelFormat != raw.pixelFormat || fx->out.width != ow || fx->out.height != oh || fx->out.pixelFormat != out_pf) { NSUInteger key[6] = {cw, ch, raw.pixelFormat, ow, oh, out_pf}; @@ -11671,7 +11790,7 @@ static int fx_wanted(MTLViewport vp, NSUInteger cw, NSUInteger ch) { [ce copyFromTexture:raw sourceSlice:0 sourceLevel:0 sourceOrigin:MTLOriginMake(r[0], r[1], 0) sourceSize:MTLSizeMake(cw, ch, 1) toTexture:fx->in destinationSlice:0 destinationLevel:0 destinationOrigin:MTLOriginMake(0, 0, 0)]; [ce barrierAfterStages:MTLStageBlit beforeQueueStages:MTLStageAll visibilityOptions:resource_vis()]; // the scaler's encoders come after the copy [ce endEncoding]; - fx->sc.colorTexture = fx->in; fx->sc.inputContentWidth = cw; fx->sc.inputContentHeight = ch; fx->sc.outputTexture = fx->out; + fx->sc.colorTexture = fx->in; fx->sc.inputContentWidth = cw; fx->sc.inputContentHeight = ch; fx->sc.outputTexture = fx->out; fx->from[0] = r[0]; fx->from[1] = r[1]; [fx->sc encodeToCommandBuffer:cb]; return fx->out; } @@ -11694,12 +11813,19 @@ static int present_encode_m4(id cb, id src, id hx = snap_excess(src, dst.pixelFormat); + NSUInteger sx = up ? fx->from[0] : r[0], sy = up ? fx->from[1] : r[1], sw = up ? fx->in.width : r[2], sh = up ? fx->in.height : r[3]; + float ex[4] = {(float)sx / src.width, (float)sy / src.height, (float)sw / src.width, (float)sh / src.height}; + [g_present_args setTexture:tex.gpuResourceID atIndex:0]; [g_present_args setTexture:(hx ? hx : tex).gpuResourceID atIndex:1]; [g_present_args setSamplerState:g_blit_samp.gpuResourceID atIndex:0]; float *uv = uvb.contents; NSUInteger tw = tex.width, th = tex.height, cx = r[0], cy = r[1], cw = r[2], ch = r[3]; uv[0] = (float)cx / tw; uv[1] = (float)cy / th; uv[2] = (float)cw / tw; uv[3] = (float)ch / th; uv[4] = (cx + 0.5f) / tw; uv[5] = (cy + 0.5f) / th; uv[6] = (cx + cw - 0.5f) / tw; uv[7] = (cy + ch - 0.5f) / th; - if (atomic_load_explicit(&g_origin_ll, memory_order_relaxed)) { uv[1] += uv[3]; uv[3] = -uv[3]; } // lower-left origin: the window image is stored bottom-up + uv[8] = present_mode(src, dst.pixelFormat); uv[9] = hx != nil; uv[10] = uv[11] = 0; // of the picture before MetalFX, whose output keeps its encoding + memcpy(uv + 12, ex, sizeof ex); + if (atomic_load_explicit(&g_origin_ll, memory_order_relaxed)) { uv[1] += uv[3]; uv[3] = -uv[3]; uv[13] += uv[15]; uv[15] = -uv[15]; } // lower-left origin: the window image is stored bottom-up [g_present_args setAddress:uvb.gpuAddress atIndex:0]; [en setViewport:vp]; [en setRenderPipelineState:g_blit_pso]; [en drawPrimitives:MTLPrimitiveTypeTriangle vertexStart:0 vertexCount:3]; @@ -11710,7 +11836,7 @@ static int present_encode_m4(id cb, id src, id src, const int crop[4], id out, void (^done)(int)) { id al = [g_dev newCommandAllocator]; id cb = [g_dev newCommandBuffer]; - id uvb = [g_dev newBufferWithLength:sizeof(float) * 8 options:MTLResourceStorageModeShared]; + id uvb = [g_dev newBufferWithLength:sizeof(float) * 16 options:MTLResourceStorageModeShared]; if (!al || !cb || !uvb) { MLOG("[RED] offscreen present: no command allocator or buffer"); return 0; } id raw = present_raw_src(src); fx_state fx = {0}; @@ -11751,6 +11877,7 @@ static void present_gpu_note(double s, int fx) { static int present_now(const tex_t *t, double period, uint64_t frame, uint64_t ticket, const int crop[4], id supplied, double target_ts) { if (g_present_stall_ns) { struct timespec ts = {0, (long)g_present_stall_ns}; nanosleep(&ts, NULL); } g_present_n++; + hdr_update(); if (bootprof() && (g_present_n == 1 || g_present_n % 120 == 0)) { if (g_present_n == 1) g_bp_first_present = bp_now() - g_bp_t0; MLOG("[bootprof] t=%.3f s present #%llu: metallibs loaded %llu / %.1f ms; pipelines created %llu / %.1f ms (all synchronous on the submit thread); prewarmed and used: shaders %llu / render pipelines %llu / compute pipelines %llu; " @@ -12117,7 +12244,13 @@ uint64_t mtl_present_ext(uint64_t serial, double period, void *ev, uint64_t ev_v // The ring's length is the number of frames the game may run ahead of the screen: DM_PRESENT_SNAP= (2 to SNAP_MAX; 1 = 2). Off by default; the iOS launcher's Frame buffering toggle enables it. #define SNAP_MAX 4 // old: the slot's texture before the presented size or format last changed, resident until the slot's next use (snap_take) -static struct { id tex, old; uint64_t serial; _Atomic uint64_t ticket, ev; } g_snap[SNAP_MAX]; +static struct { id tex, old, hx, hx_old; uint64_t serial; _Atomic uint64_t ticket, ev; _Atomic int hx_on; } g_snap[SNAP_MAX]; // hx, hx_on: HDR output's highlights (hdr_merge) +// The highlights to add to a present of the ring slot src into a target of format dst: nil unless HDR output wrote them for that slot's copy and the target is extended-range +static id snap_excess(id src, MTLPixelFormat dst) { + if (dst != MTLPixelFormatRGBA16Float || !src) return nil; + for (int i = 0; i < SNAP_MAX; i++) if (g_snap[i].tex == src) return atomic_load(&g_snap[i].hx_on) ? g_snap[i].hx : nil; + return nil; +} static id g_snap_alloc[SNAP_MAX]; static id g_snap_cb[SNAP_MAX]; static id g_snap_res; static id g_snap_ev; static uint64_t g_snap_v; static _Atomic uint64_t g_snap_next, g_snap_cmt; // g_snap_cmt: the last copy the encode thread has committed (its g_snap_v) @@ -12126,6 +12259,7 @@ int mtl_snap_on(void) { static int v = -1; if (v >= 0) return v; const char *e = getenv("DM_PRESENT_SNAP"); int n = e ? atoi(e) : DM_GAME_PRESENT_SNAP ? 2 : 0; // the profile's present_snap (a ring of 2) + if (n <= 0 && mtl_hdr_on()) { n = 2; MLOG("present snapshots: on for HDR output, whose copy writes the highlights (hdr_merge)"); } v = n > 0; g_snap_n = n < 2 ? 2 : n > SNAP_MAX ? SNAP_MAX : n; if (v) MLOG("present snapshots (DM_PRESENT_SNAP=%d): each present is copied into a ring of %d host textures and the game's texture is free at once", n, g_snap_n); return v; @@ -12151,6 +12285,46 @@ uint64_t mtl_snap_acquire_wait(void) { if (ws_on()) ws_put(WS_SNAP, w0, blocked); return ws_t0() - w0; } +// HDR output (see hdr_drawn): the copy into a ring slot as a draw: the swapchain's bytes as they are (dst, 8 bits like the blit's copy), and beside them (hx, RG11B10Float) the +// side texture's excess where the picture's luminance, decoded from sRGB, is still what the tone mapping wrote (within 1.2 % + 5 %, fading out over as much again: the AA pass and +// the upscale move it a little, the HUD, fades and later effects move it more; dm/gfx/msl/hdr_merge.metal). The pass sits between the same queue barriers as the blit it replaces +static id g_hm_lib; static id g_hm_pso; static MTLPixelFormat g_hm_fmt; static id g_hm_args; static id g_hm_buf; static id g_hm_samp; +static int hdr_merge(id cb, id src, id dst, id hx, int slot) { + NSError *e = nil; + if (!g_hm_lib) { + g_hm_lib = host_lib("hdr_merge", &e); + MTL4ArgumentTableDescriptor *ad = [MTL4ArgumentTableDescriptor new]; ad.maxTextureBindCount = 2; ad.maxSamplerStateBindCount = 1; ad.maxBufferBindCount = 1; + g_hm_args = [g_dev newArgumentTableWithDescriptor:ad error:&e]; + g_hm_buf = [g_dev newBufferWithLength:256 * SNAP_MAX options:MTLResourceStorageModeShared]; + MTLSamplerDescriptor *sd = [MTLSamplerDescriptor new]; sd.minFilter = sd.magFilter = MTLSamplerMinMagFilterLinear; sd.supportArgumentBuffers = YES; + g_hm_samp = [g_dev newSamplerStateWithDescriptor:sd]; + if (!g_hm_lib || !g_hm_args || !g_hm_buf || !g_hm_samp) MDIE("HDR: the present copy's pipeline: %s", e.localizedDescription.UTF8String); + res_add(g_hm_buf); res_commit(); + } + if (!g_hm_pso || g_hm_fmt != dst.pixelFormat) { + MTLRenderPipelineDescriptor *pd = [MTLRenderPipelineDescriptor new]; + pd.vertexFunction = [g_hm_lib newFunctionWithName:@"vs"]; pd.fragmentFunction = [g_hm_lib newFunctionWithName:@"fs"]; + pd.colorAttachments[0].pixelFormat = dst.pixelFormat; pd.colorAttachments[1].pixelFormat = MTLPixelFormatRG11B10Float; + g_hm_pso = [g_dev newRenderPipelineStateWithDescriptor:pd error:&e]; g_hm_fmt = dst.pixelFormat; + if (!g_hm_pso) MDIE("HDR: the present copy's pipeline: %s", e.localizedDescription.UTF8String); + } + float *m = (float *)((uint8_t *)g_hm_buf.contents + 256 * slot); + m[0] = (float)g_hdr_rw / HDR_SIDE_W; m[1] = (float)g_hdr_rh / HDR_SIDE_H; m[2] = m[3] = 0; + MTL4RenderPassDescriptor *rp = [MTL4RenderPassDescriptor new]; + rp.colorAttachments[0].texture = dst; rp.colorAttachments[0].loadAction = MTLLoadActionDontCare; rp.colorAttachments[0].storeAction = MTLStoreActionStore; + rp.colorAttachments[1].texture = hx; rp.colorAttachments[1].loadAction = MTLLoadActionDontCare; rp.colorAttachments[1].storeAction = MTLStoreActionStore; + id en = [cb renderCommandEncoderWithDescriptor:rp]; + if (!en) return 0; + [en barrierAfterQueueStages:MTLStageAll beforeStages:MTLStageAll visibilityOptions:resource_vis()]; // after the frame's work and the slot's last blit to the screen + [en setArgumentTable:g_hm_args atStages:MTLRenderStageFragment]; + [g_hm_args setTexture:src.gpuResourceID atIndex:0]; [g_hm_args setTexture:g_hdr_side.gpuResourceID atIndex:1]; + [g_hm_args setSamplerState:g_hm_samp.gpuResourceID atIndex:0]; [g_hm_args setAddress:g_hm_buf.gpuAddress + 256 * slot atIndex:0]; + [en setRenderPipelineState:g_hm_pso]; + [en drawPrimitives:MTLPrimitiveTypeTriangle vertexStart:0 vertexCount:3]; + [en barrierAfterStages:MTLStageFragment beforeQueueStages:MTLStageAll visibilityOptions:resource_vis()]; // the game's next writes to its texture come after the copy + [en endEncoding]; + return 1; +} // Copy the presented texture into the next ring slot; returns the slot's serial (0 = no snapshot: the game's texture is presented as before) static uint64_t snap_take(const tex_t *t) { int k = (int)(atomic_load(&g_snap_next) % (uint64_t)g_snap_n); @@ -12160,6 +12334,7 @@ static uint64_t snap_take(const tex_t *t) { // The texture this slot held before it was last replaced: every present that read it had been committed when the replacing copy was (the busy check then), and the queue // signals g_snap_ev only after all work committed before the signal, so those presents have run and it can leave the residency set if (g_snap[k].old) { [g_snap_res removeAllocation:g_snap[k].old]; [g_snap_res commit]; g_snap[k].old = nil; } + if (g_snap[k].hx_old) { [g_snap_res removeAllocation:g_snap[k].hx_old]; [g_snap_res commit]; g_snap[k].hx_old = nil; } id src = tex_mtl(t); if (!src) return 0; // The copy reads the source before present_now runs, so it needs the same commit as present_now: a window texture presented before any submission has used it was made just // now and is outside the committed residency set. DRAGON QUEST XI S presents three times before its first submission; on an A17 Pro the copy faulted silently (its command buffer @@ -12176,9 +12351,12 @@ static uint64_t snap_take(const tex_t *t) { } } id dst = g_snap[k].tex; - if (!dst || dst.width != src.width || dst.height != src.height || dst.pixelFormat != src.pixelFormat) { + // HDR output: the copy is hdr_merge's draw when the tone mapping drew in this frame and the peak is above SDR white (a slot that is also a render target), else the blit + int hdr = mtl_hdr_on() && src.sampleCount == 1 && src.textureType == MTLTextureType2D && (src.pixelFormat == MTLPixelFormatRGBA8Unorm || src.pixelFormat == MTLPixelFormatBGRA8Unorm); + int ex = hdr && g_hdr_drawn && mtl_hdr_peak() > 1.0001f; if (hdr) g_hdr_drawn = 0; + if (!dst || dst.width != src.width || dst.height != src.height || dst.pixelFormat != src.pixelFormat || (hdr && !(dst.usage & MTLTextureUsageRenderTarget))) { MTLTextureDescriptor *d = [MTLTextureDescriptor texture2DDescriptorWithPixelFormat:src.pixelFormat width:src.width height:src.height mipmapped:NO]; - d.storageMode = MTLStorageModePrivate; d.usage = MTLTextureUsageShaderRead; + d.storageMode = MTLStorageModePrivate; d.usage = MTLTextureUsageShaderRead | (hdr ? MTLTextureUsageRenderTarget : 0); // The old texture stays resident (and referenced) until the slot's next use: the busy check above means its last present was committed, not that the GPU has run it, // and a texture removed here was sampled outside the residency set by a display command buffer still queued g_snap[k].old = dst; @@ -12188,14 +12366,29 @@ static uint64_t snap_take(const tex_t *t) { serial_put(g_snap[k].serial, dst); g_snap[k].tex = dst; MLOG("present snapshots: ring slot %d %lux%lu format %lu", k, (unsigned long)dst.width, (unsigned long)dst.height, (unsigned long)dst.pixelFormat); } + id hx = g_snap[k].hx; + if (ex && (!hx || hx.width != src.width || hx.height != src.height)) { + MTLTextureDescriptor *d = [MTLTextureDescriptor texture2DDescriptorWithPixelFormat:MTLPixelFormatRG11B10Float width:src.width height:src.height mipmapped:NO]; + d.storageMode = MTLStorageModePrivate; d.usage = MTLTextureUsageShaderRead | MTLTextureUsageRenderTarget; + g_snap[k].hx_old = hx; // resident until the slot's next use, as the slot's own texture (old above) + g_snap[k].hx = hx = [g_dev newTextureWithDescriptor:d]; + if (hx) { hx.label = @"HDR highlights"; [g_snap_res addAllocation:hx]; MLOG("present snapshots: ring slot %d HDR highlights %lux%lu", k, (unsigned long)hx.width, (unsigned long)hx.height); } + else { RED_ONCE(142, "HDR: no highlights texture for the present copy; presenting without highlights"); ex = 0; } + [g_snap_res commit]; + } + atomic_store(&g_snap[k].hx_on, ex); [g_snap_alloc[k] reset]; id cb = g_snap_cb[k]; [cb beginCommandBufferWithAllocator:g_snap_alloc[k]]; + if (ex) { if (!hdr_merge(cb, src, dst, hx, k)) { atomic_store(&g_snap[k].hx_on, 0); [cb endCommandBuffer]; return 0; } } + else { id ce = [cb computeCommandEncoder]; [ce barrierAfterQueueStages:MTLStageAll beforeStages:MTLStageBlit visibilityOptions:resource_vis()]; // after the frame's work and the slot's last blit to the screen [ce copyFromTexture:src sourceSlice:0 sourceLevel:0 sourceOrigin:MTLOriginMake(0, 0, 0) sourceSize:MTLSizeMake(src.width, src.height, 1) toTexture:dst destinationSlice:0 destinationLevel:0 destinationOrigin:MTLOriginMake(0, 0, 0)]; [ce barrierAfterStages:MTLStageBlit beforeQueueStages:MTLStageAll visibilityOptions:resource_vis()]; // the game's next writes to its texture come after the copy - [ce endEncoding]; [cb endCommandBuffer]; + [ce endEncoding]; + } + [cb endCommandBuffer]; uint64_t v = ++g_snap_v; if (rec_on() && g_rb_started) { // a block of the encode thread's queue, behind the frame's work (see above) if (on_sub_thread()) rec_publish(); // what the submitting thread recorded outside a submission goes first diff --git a/dm/host/front_mac.m b/dm/host/front_mac.m index 714987f32..0315201f8 100644 --- a/dm/host/front_mac.m +++ b/dm/host/front_mac.m @@ -1196,6 +1196,14 @@ void front_run(void (*guest)(void)) { g_layer = [CAMetalLayer layer]; g_layer.device = MTLCreateSystemDefaultDevice(); g_layer.pixelFormat = MTLPixelFormatBGRA8Unorm; + int mtl_hdr_on(void); void mtl_hdr_headroom(float h); + if (mtl_hdr_on()) { // HDR output (nvn_mtl.m, HDR output): an extended-range layer, 1.0 = SDR white, and the screen's headroom for the tone mapping's peak, read four times a second + g_layer.pixelFormat = MTLPixelFormatRGBA16Float; g_layer.wantsExtendedDynamicRangeContent = YES; + CGColorSpaceRef cs = CGColorSpaceCreateWithName(kCGColorSpaceExtendedLinearSRGB); g_layer.colorspace = cs; CGColorSpaceRelease(cs); + NSScreen *sc = win.screen ?: NSScreen.mainScreen; + LOG("front end: HDR layer (RGBA16Float, extended linear sRGB); screen headroom now %.2f, at most %.2f", sc.maximumExtendedDynamicRangeColorComponentValue, sc.maximumPotentialExtendedDynamicRangeColorComponentValue); + [NSTimer scheduledTimerWithTimeInterval:0.25 repeats:YES block:^(NSTimer *tm) { (void)tm; mtl_hdr_headroom((float)(win.screen ?: NSScreen.mainScreen).maximumExtendedDynamicRangeColorComponentValue); }]; + } g_layer.contentsScale = win.backingScaleFactor; // Apple's HUD over the game (frame rate, frame and GPU time, memory): front_hud_env turns it on before the first device. The layer's developerHUDProperties // stay unset: mode=default forces this layer's HUD on, so the Metal HUD menu's Hide HUD left it on screen diff --git a/dm/host/hle_hid.c b/dm/host/hle_hid.c index 6084ae7f2..2eefbccfd 100644 --- a/dm/host/hle_hid.c +++ b/dm/host/hle_hid.c @@ -207,6 +207,7 @@ void hid_keys_init(void) { // saves a host texture's level 0 raw, every slice (a cube's six faces) (nvn_mtl.m dm_gfx_texraw), "drag [seconds=0.3]" moves one finger in a straight line between the two normalized picture // points over that time and lifts it (the touch screen's pans and swipes, hle_touch.c gesture_sample), "dump" logs what each guest thread waits on and the graphics wait points // (crash_dump_now, as the iOS app's DM_LOG_PORT dump; scripts/regress.py asks for it before stopping a hung game), "#…" is ignored. +// "hdrpeak " sets the HDR peak in SDR whites (nvn_mtl.m mtl_hdr_set_peak; 1 = the game's own picture, 0 = the screen's headroom again), "hdrknee " the SDR value its curve leaves the game's at, // Look at a screen, decide, press, look again: menus whose timing moves from run to run (loading) need no blind timed script static void *ctl_main(void *arg) { const char *path = arg; long off = 0; char line[1100]; @@ -229,6 +230,8 @@ static void *ctl_main(void *arg) { if (!strncmp(line, "env ", 4)) { char *eq = strchr(line + 4, '='); if (eq) { *eq = 0; setenv(line + 4, eq + 1, 1); LOG("DM_CTL: env %s=%s", line + 4, eq + 1); } continue; } if (!strncmp(line, "texdump ", 8)) { void dm_gfx_texdump_next(const char *path); dm_gfx_texdump_next(line + 8); LOG("DM_CTL: colour targets of the next frame → %s (as DM_NVN_TEXDUMP)", line + 8); continue; } if (!strncmp(line, "passdump ", 9)) { void dm_gfx_passdump_next(long n); dm_gfx_passdump_next(atol(line + 9)); LOG("DM_CTL: passes of the next %ld frames (as DM_NVN_PASSDUMP)", atol(line + 9)); continue; } + if (!strncmp(line, "hdrpeak ", 8)) { void mtl_hdr_set_peak(float p); mtl_hdr_set_peak(strtof(line + 8, NULL)); LOG("DM_CTL: HDR peak %s", line + 8); continue; } + if (!strncmp(line, "hdrknee ", 8)) { void mtl_hdr_set_knee(float k); mtl_hdr_set_knee(strtof(line + 8, NULL)); LOG("DM_CTL: HDR knee %s", line + 8); continue; } if (!strncmp(line, "drag ", 5)) { // one finger, a step every 8 ms float x0, y0, x1, y1, s = 0.3f; if (sscanf(line + 5, "%f %f %f %f %f", &x0, &y0, &x1, &y1, &s) < 4) { LOG("[RED] DM_CTL drag wants x0 y0 x1 y1 [seconds]"); continue; } int n = (int)(s / 0.008f); if (n < 2) n = 2; diff --git a/dm/host/tests/t_hid.c b/dm/host/tests/t_hid.c index 11ad41364..f751cb6b1 100644 --- a/dm/host/tests/t_hid.c +++ b/dm/host/tests/t_hid.c @@ -46,6 +46,8 @@ void dm_ctl_cvar_i32(uint64_t off, int32_t v) { (void)off; (void)v; } void dm_ctl_cvar_f32(uint64_t off, float v) { (void)off; (void)v; } void dm_gfx_texdump_next(const char *path) { (void)path; } void dm_gfx_passdump_next(long n) { (void)n; } +void mtl_hdr_set_peak(float p) { (void)p; } +void mtl_hdr_set_knee(float k) { (void)k; } static int applet_depth, applet_opens, applet_closes_at_depth0; // hle_oe.c: the applet stand-ins open and close their applet around the answer void dm_applet_enter(const char *name) { (void)name; applet_depth++; applet_opens++; } void dm_applet_leave(void) { applet_closes_at_depth0 += --applet_depth == 0; } diff --git a/dm/ios/Sources/front_ios.m b/dm/ios/Sources/front_ios.m index b941e7772..f9afbde1c 100644 --- a/dm/ios/Sources/front_ios.m +++ b/dm/ios/Sources/front_ios.m @@ -26,6 +26,8 @@ static void front_ios_power_watch(void); static void front_ios_audio_watch(void); static UILabel *front_ios_fps_label(void); +int mtl_hdr_on(void); // nvn_mtl.m: HDR output (the profile's hdr, DM_HDR) +void mtl_hdr_headroom(float h); @interface DmGameView : UIView @end @@ -35,6 +37,10 @@ - (instancetype)initWithFrame:(CGRect)f { if ((self = [super initWithFrame:f])) { CAMetalLayer *l = (CAMetalLayer *)self.layer; l.device = MTLCreateSystemDefaultDevice(); l.pixelFormat = MTLPixelFormatBGRA8Unorm; + if (mtl_hdr_on()) { // HDR output (nvn_mtl.m, HDR output): an extended-range layer, 1.0 = SDR white; values above it light the screen's EDR headroom + l.pixelFormat = MTLPixelFormatRGBA16Float; l.wantsExtendedDynamicRangeContent = YES; + CGColorSpaceRef cs = CGColorSpaceCreateWithName(kCGColorSpaceExtendedLinearSRGB); l.colorspace = cs; CGColorSpaceRelease(cs); + } l.contentsScale = UIScreen.mainScreen.nativeScale; l.framebufferOnly = NO; l.opaque = YES; self.backgroundColor = UIColor.blackColor; } @@ -99,7 +105,18 @@ - (void)startGame { front_hud_env(); // before the process's first Metal device, which reads MTL_HUD_ENABLED (an env file's DM_METAL_HUD sets it in main.m already; GitHub issue #840) const char *dev = front_ios_device_check(); // that first device: a GPU without BC textures is refused here, not at launch, so the launcher's HUD switch counts if (dev) { [self showHint:[NSString stringWithFormat:@"This device is not supported.\n%s", dev]]; return; } // the host does not start + // HDR output needs a screen with EDR headroom: without it the game keeps its SDR targets, whatever the launcher says (DM_HDR=2 keeps it on, to test the layer). Decided before the + // game view, whose layer follows it, and before the host's graphics start (mtl_hdr_on reads it once) + UIScreen *scr = self.view.window.screen ?: UIScreen.mainScreen; + const char *he = getenv("DM_HDR"); + if (scr.potentialEDRHeadroom <= 1.0 && !(he && atoi(he) == 2)) { setenv("DM_HDR", "0", 1); LOG("front end: this screen has no EDR headroom (potential %.2f): HDR output off", scr.potentialEDRHeadroom); } self.game =[[DmGameView alloc] initWithFrame:self.view.bounds]; + if (mtl_hdr_on()) { // the tone mapping's peak follows the headroom the screen has now (it changes with the brightness), read four times a second + LOG("front end: HDR layer (RGBA16Float, extended linear sRGB); EDR headroom now %.2f, at most %.2f", scr.currentEDRHeadroom, scr.potentialEDRHeadroom); + __weak DmRootVC *ws = self; + [NSTimer scheduledTimerWithTimeInterval:0.25 repeats:YES block:^(NSTimer *t) { DmRootVC *vc = ws; if (!vc) { [t invalidate]; return; } + mtl_hdr_headroom((float)(vc.view.window.screen ?: UIScreen.mainScreen).currentEDRHeadroom); }]; + } [self.view addSubview:self.game]; g_layer = (CAMetalLayer *)self.game.layer; if (getenv("DM_METAL_HUD") && atoi(getenv("DM_METAL_HUD"))) { g_layer.developerHUDProperties = @{@"mode": @"default"}; LOG("front end: Metal performance HUD on (DM_METAL_HUD)"); } // Apple's HUD over the game: frame rate, frame and GPU time, memory diff --git a/dm/tools/docs_patches.py b/dm/tools/docs_patches.py index f758427f5..be1cc2c67 100644 --- a/dm/tools/docs_patches.py +++ b/dm/tools/docs_patches.py @@ -113,6 +113,8 @@ def payoff(gap): # the Gaps order: open gaps, then blocked, kept and unreviewe 'why': f'what iOS does for every game: {DEFAULT_SOON}'}), ('present_snap', '', {'what': 'present snapshots', 'now': 'game', 'goal': 'host', 'purpose': 'rendering', 'why': f'a setting the game was tested with: {DEFAULT_SOON}'}), ('pso_mtl4', '', {'what': 'MTL4Compiler pipelines on iOS', 'now': 'game', 'goal': 'host', 'purpose': 'rendering', 'why': f'a setting the game was tested with on an iPhone: {DEFAULT_SOON}'}), + ('hdr', '', {'what': "HDR output: the game's tone mapping shader replaced by the profile's (DM_HDR)", 'now': 'game', 'goal': 'game', 'purpose': 'rendering', + 'why': "the presenting, the layer and the screen's headroom are the host's, but where a game tone maps and how its curve is raised are its own shader's"}), ('game_env', 'DOTNET_GCRegionRange=', {'what': ".NET's garbage-collector region range", 'now': 'game', 'goal': 'game', 'purpose': 'address space', 'why': "measured for the game's regions"}), ('stdalloc_cap', '', {'what': "StandardAllocator's address space", 'now': 'game', 'goal': 'game', 'purpose': 'address space', 'why': "what the regions after it need, measured by the NSP script's region check; the host's cap is the default"}), diff --git a/docs/agents/game-patches.md b/docs/agents/game-patches.md index 2cdc1aa78..71eebef44 100644 --- a/docs/agents/game-patches.md +++ b/docs/agents/game-patches.md @@ -621,6 +621,7 @@ Profile keys (`game.tsv`) whose value changes what the host does, set away from |---|---|---|---| | `enc_thread on` | [Need For Speed™ Hot Pursuit Remastered](../../dm/games/010029B0118E8000/README.md) | `game` → `host`, gap | what iOS does for every game: a default for every game after `regress.py check` on the games | | `game_env DOTNET_GCRegionRange=60000000` | [Absolum](../../dm/games/0100A83022A22000/README.md) | `game`, at goal | measured for the game's regions | +| `hdr f7ee6fe10b925543 hdr_tonemap.metal` | [The Legend of Zelda: Breath of the Wild](../../dm/games/01007EF00011E000/README.md) | `game`, at goal | the presenting, the layer and the screen's headroom are the host's, but where a game tone maps and how its curve is raised are its own shader's | | `present_snap on` | [The Legend of Zelda: Breath of the Wild](../../dm/games/01007EF00011E000/README.md) | `game` → `host`, gap | a setting the game was tested with: a default for every game after `regress.py check` on the games | | `pso_mtl4 on` | [The Legend of Zelda: Breath of the Wild](../../dm/games/01007EF00011E000/README.md) | `game` → `host`, gap | a setting the game was tested with on an iPhone: a default for every game after `regress.py check` on the games | | `shader_key_skip 0x770+0x78` | [Batman: Arkham Knight](../../dm/games/0100ACD0163D0000/README.md) | `game`, at goal | the engine shares one control section among shaders; leaving GLSLC's compile information out of every game's keys would change every shader key | diff --git a/docs/env_switches.tsv b/docs/env_switches.tsv index bf04b4f4b..d2b045892 100644 --- a/docs/env_switches.tsv +++ b/docs/env_switches.tsv @@ -57,6 +57,12 @@ DM_GPU_CAPTURE diagnostic `=[:]` or `gpu:[:]` wr DM_GPU_CAPTURE_DIR path Folder for `DM_GPU_CAPTURE`'s traces (default `gpu_captures/` in the working directory) DM_GPU_CAPTURE_KEEP diagnostic Before each GPU capture, prunes the capture folder's `dm_*.gputrace` entries so that the newest `KEEP - 1` stay (default 2) DM_GUEST_SPAWN_TEST diagnostic `=1` runs a self-test before `nnMain`: a host-started guest thread does SDK malloc / free and locks an `nn::os` mutex, and the result is logged; `=2` also keeps a resident thread +DM_HDR player `0` turns HDR output off for a game whose profile has `hdr` (on by default there): the profile's tone mapping shader also writes its highlights above SDR white, which the present adds where the 3D picture shows, into an extended-range layer, and turns present snapshots on (an explicit `DM_PRESENT_SNAP=0` turns HDR output off: their copy carries the highlights); the iOS app sets `0` on a screen without EDR headroom, and `2` keeps it on there (to test the layer) +DM_HDR_GAIN switch `=` scales HDR output's highlights (default 1) +DM_HDR_KNEE switch `=`: the SDR value where HDR output's tone curve leaves the game's (0.05 to 0.95, default 0.6); the Mac's `DM_CTL` `hdrknee ` sets it while the game runs +DM_HDR_MAX switch `=`: the highest peak HDR output follows the screen's headroom to, in SDR whites (default 8) +DM_HDR_PEAK switch `=` fixes HDR output's peak in SDR whites instead of following the screen's headroom; the Mac's `DM_CTL` `hdrpeak ` sets it while the game runs (`0`: back to the headroom) +DM_HDR_SHOT_RAW diagnostic `1`: with HDR output, a screenshot (`DM_NVN_SHOT`, `DM_CTL` `shot`) also writes the picture as the present draws it into an extended-range layer, highlights included, to `.rgba16f` (a line `RGBA16F `, then the rows as half floats, linear, 1 = SDR white), at `DM_NVN_SHOT_OUT`'s size when set DM_HID_LOG diagnostic `1` logs the HID button usage number of each button pressed on a raw HID gamepad (`pad_hid_mac.h`), to write a `DM_HID_MAP` DM_HID_MAP switch `=` such as `A=1,B=2,X=4,Y=5`: which HID button usage presses each Switch button on a raw HID gamepad (names A B X Y LS RS L R ZL ZR PLUS MINUS LEFT UP RIGHT DOWN; default 8BitDo's D-input numbering by position, `pad_hid.h`); Mac only DM_HID_PADS switch `0` stops reading raw HID gamepads that GameController does not list, such as an 8BitDo in its own mode (`pad_hid_mac.h`); Mac only diff --git a/docs/settings.md b/docs/settings.md index 0f992d4ca..7e4acc698 100644 --- a/docs/settings.md +++ b/docs/settings.md @@ -3,7 +3,7 @@ The screen before the game starts shows the settings the game supports. A setting that only matters while another is on appears under it and disappears when that one is turned off. Tap **Play** to start; the app remembers your choices for the next start. All settings are on one scrollable screen, under **Game**, **Controls** and **Performance**. Tap a menu or move a slider there; no settings page needs opening. **Upscaling** is in **Game**, beside **Console mode**. A Mac app shows the same settings in a [window of its own](#the-mac-settings-window). - [Tested defaults](#tested-defaults) -- [Game](#game): [Frame rate](#frame-rate), [Render scale](#render-scale), [Console mode](#console-mode), [Upscaling](#upscaling), [Game language](#game-language), [Frame-rate counter](#frame-rate-counter) +- [Game](#game): [Frame rate](#frame-rate), [Render scale](#render-scale), [Console mode](#console-mode), [Upscaling](#upscaling), [Game language](#game-language), [Frame-rate counter](#frame-rate-counter), [HDR](#hdr) - [Controls](#controls): [Touch feedback](#touch-feedback), [Game rumble](#game-rumble), [Rumble strength](#rumble-strength), [Rumble cutoff](#rumble-cutoff) - [Performance](#performance): [Frame buffering](#frame-buffering), [Metal 4 pipelines](#metal-4-pipelines), [Pipeline prebuild](#pipeline-prebuild), [Encode thread](#encode-thread), [Hold frames when late](#hold-frames-when-late), [Performance HUD](#performance-hud), [Reset to tested defaults](#reset-to-tested-defaults) - [The Mac settings window](#the-mac-settings-window): [Display mode](#display-mode) @@ -59,6 +59,14 @@ The game's own languages. The app was packed with one of them; pick another here Shows the frames per second at the top left. +### HDR + +On a screen with HDR (an iPhone, or a Mac with an XDR or HDR display), a game whose profile supports it (Breath of the Wild) shows its brightest parts, the sun, the sky around it, sunlit grass, fire and reflections, brighter than normal white, up to what the screen can show at its current brightness. Everything darker, the HUD, the menus and the map look as before. On by default where it is offered; the app turns it off by itself on a screen without HDR. + +- **When to change it:** turn it off to compare with the game's own picture, or to save a little power. +- **Cost:** the game's picture is copied with its highlights each frame and they are added when it is shown: a little GPU time and about 40 MB of memory. It needs [Frame buffering](#frame-buffering): the highlights travel with its copy of each frame, so with Frame buffering off the picture is the game's own. Bright highlights also make an OLED screen draw more power. +- **How bright:** the highlights follow the headroom the screen reports, which grows as the screen's brightness is turned down; `DM_HDR_MAX` (8 by default) caps it and `DM_HDR_PEAK` fixes it, in multiples of normal white ([Settings in a file](#settings-in-a-file)). + ## Controls ### Touch feedback