Skip to content

Commit c03baef

Browse files
LinuxJessiclaude
andcommitted
Scene archives: the "undecodable XTX variant" was ARX compression
The ~500 embedded-texture rows browse reported as undecodable (all in scene/cf*.a) were packed members: the archives keep their members ARX-compressed in place, and the coder passes literal words through, so a packed XTX's header survives inside the stream while its pixels do not exist until decompression. - browse.py: iter_arx_containers() finds and decompresses every in-file ARX container (1,864 in the 112 retail scene archives: 603 XTX, 778 lex, 483 FPK); the sweep decodes the XTX members with the lex models packed beside them as palette source and skips raw hits that fall inside a container's span - embedded_textures.csv gains a `packed` column; retail run: 1,363 PNGs, 1,696 duplicates, 0 undecodable (was 528). All 603 packed textures are byte-identical copies of standalone char/enemy/obj/map .xtx, so the manifest now doubles as a per-scene provenance map - two garbage PNGs the old sweep decoded from inside compressed streams are gone (cf3021.a_208e4c, cf3070.a_61690c) - FORMATS.md: .a archive layout (TOC + 0x100 record header + ARX member); the repack note's "re-framed canvas with 4-byte inserts" is the ARX bit stream, which is why the entry-level palette sweep works - UPDATES.md / BROWSING.md / README: counts and narrative Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 parent ad26909 commit c03baef

6 files changed

Lines changed: 186 additions & 26 deletions

File tree

‎README.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -210,7 +210,8 @@ out/
210210
`dump/` is the disc as the game sees it; `browse` then builds a
211211
human-readable mirror next to it (PNG/WAV/MP4/UTF-8) — including a
212212
disc-wide sweep that pulls ~1,400 more textures out of the effect/scene/UI
213-
containers (`browse/textures_png/_embedded/`), the PS2 memory-card icon
213+
containers (`browse/textures_png/_embedded/`; scene archives are
214+
ARX-packed in place and get decompressed first), the PS2 memory-card icon
214215
resource, and string tables sniffed out of binary files. A guided tour of
215216
what's in there — including the **eight developers' personal folders that
216217
shipped on the retail disc** — is in [docs/BROWSING.md](docs/BROWSING.md).

‎browse.py‎

Lines changed: 81 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -31,9 +31,14 @@
3131
count 1-64, in-bounds header table). Effect libraries (``.esd``/``.esp``),
3232
scene archives (``.a``), battle data (``.bin``) and the NLNK/NBGL/NBXX UI
3333
containers (``.npr``/``.rbg``/``.bxx``) carry ~3,000 textures this way —
34-
more than the standalone set. Blobs byte-identical to an already-decoded
35-
texture are recorded as duplicates, not re-written; the sweep is logged
36-
per-carrier in ``browse/embedded_textures.csv``.
34+
more than the standalone set. Scene archives, ``.fpk`` packs and ``.arc``
35+
bundles keep their members **ARX-compressed in place**; those are
36+
decompressed first (``iter_arx_containers``) and the XTX inside are
37+
paletted by the ``lex`` models packed beside them, nearest member first.
38+
Blobs byte-identical to an already-decoded texture are recorded as
39+
duplicates, not re-written; the sweep is logged per-carrier in
40+
``browse/embedded_textures.csv`` (``packed`` = "arx" for members that
41+
only exist after decompression).
3742
3843
* ``audio`` — decode ``.vds``/``.vdm`` streamed audio to 16-bit WAV.
3944
@@ -689,6 +694,41 @@ def iter_embedded_xtx(data: bytes) -> Iterable[tuple[int, bytes]]:
689694
i += 4
690695

691696

697+
def iter_arx_containers(data: bytes) -> Iterable[tuple[int, int, bytes]]:
698+
"""Yield ``(offset, span, payload)`` for every ARX container inside ``data``.
699+
700+
Scene archives (``.a``), ``.fpk`` packs, ``.arc`` bundles and a few
701+
``.bin``/``.npr`` files store their members ARX-compressed in place:
702+
a 16-byte header (magic, uncompressed size, container size including
703+
the header, 0), the 30-word LUT, then the bit stream. The members —
704+
XTX textures, the ``lex`` models that bind their palettes, FPK packs,
705+
FL00 event containers — only exist after decompression. Because the
706+
ARX coder passes literal words through verbatim, the *header* of a
707+
packed XTX still reads as ``XTX\0`` + sane sizes inside the stream,
708+
which is exactly the "undecodable in-``.a`` variant" the raw sweep used
709+
to report (528 on the retail disc): callers must skip raw hits that
710+
fall inside a container's span. A hit must carry a sane header and
711+
decompress to exactly the announced size; anything else is skipped.
712+
"""
713+
i, n = 0, len(data)
714+
while True:
715+
i = data.find(arx.MAGIC, i)
716+
if i < 0:
717+
return
718+
if i + 16 <= n:
719+
usize, csize, zero = struct.unpack_from("<III", data, i + 4)
720+
if zero == 0 and usize > 0 and 16 + 30 * 4 < csize <= n - i:
721+
try:
722+
payload = arx.decompress(data[i:i + csize])
723+
except (arx.ARXError, struct.error, ValueError):
724+
payload = b""
725+
if len(payload) == usize:
726+
yield i, csize, payload
727+
i += csize
728+
continue
729+
i += 4
730+
731+
692732
# extensions that never carry embedded XTX art (movies, audio, plain media);
693733
# everything else in the dump gets swept
694734
_NO_XTX_CARRIER = (".xtx", ".pss", ".ipu", ".vds", ".vdm", ".jpg", ".jpeg", ".txt")
@@ -1136,29 +1176,55 @@ def bump(key: str, n: int = 1) -> None:
11361176
if not data:
11371177
continue
11381178
rel = src.relative_to(dump)
1179+
# members compressed in place (scene archives, packs) come first:
1180+
# their XTX only exist after decompression, and the literal
1181+
# header words inside a compressed stream must not be taken for
1182+
# raw hits (that was the old "undecodable in-.a variant")
1183+
containers = list(iter_arx_containers(data))
1184+
spans = [(o, o + span) for o, span, _ in containers]
1185+
lex_pool = [(o, pay) for o, _, pay in containers
1186+
if pay[:4] == b"lex\x00"]
1187+
# (offset label, png stem, blob, lex models, packed?)
1188+
hits: list[tuple[str, str, bytes, list[bytes], bool]] = []
11391189
for off, blob in iter_embedded_xtx(data):
1190+
if any(a <= off < b for a, b in spans):
1191+
continue
1192+
hits.append((f"0x{off:x}", f"{off:06x}", blob, [], False))
1193+
for o, _, pay in containers:
1194+
bump("textures_embedded_arx")
1195+
# a scene's textures are paletted by the lex models packed
1196+
# beside them: nearest member first (its material 0 seeds
1197+
# the base palette), the rest merged as companions
1198+
lexes = [lp for _, lp in
1199+
sorted(lex_pool, key=lambda t: abs(t[0] - o))]
1200+
for k, blob in iter_embedded_xtx(pay):
1201+
label = f"0x{o:x}" + (f"+0x{k:x}" if k else "")
1202+
stem = f"{o:06x}" + (f"_{k:x}" if k else "")
1203+
hits.append((label, stem, blob, lexes, True))
1204+
for label, stem, blob, lexes, packed in hits:
1205+
pk = "arx" if packed else ""
11401206
sha = hashlib.sha1(blob).hexdigest()
11411207
dup = seen.get(sha)
11421208
if dup is not None:
11431209
bump("textures_embedded_dup")
1144-
rows.append([str(rel), f"0x{off:x}", len(blob), sha[:12],
1145-
"", dup])
1210+
rows.append([str(rel), label, len(blob), sha[:12],
1211+
"", dup, pk])
11461212
continue
1147-
decoded = decode_xtx(blob)
1213+
decoded = decode_xtx(blob, lexes)
11481214
if decoded is None:
11491215
bump("textures_embedded_undecodable")
1150-
rows.append([str(rel), f"0x{off:x}", len(blob), sha[:12],
1151-
"", "(undecodable)"])
1216+
rows.append([str(rel), label, len(blob), sha[:12],
1217+
"", "(undecodable)", pk])
11521218
continue
11531219
w, h, rgba, source = decoded
1154-
dest = edir / rel.parent / f"{rel.name}_{off:06x}.png"
1220+
dest = edir / rel.parent / f"{rel.name}_{stem}.png"
11551221
dest.parent.mkdir(parents=True, exist_ok=True)
11561222
write_png(dest, w, h, rgba)
11571223
seen[sha] = str(dest.relative_to(browse))
11581224
bump("textures_embedded_png")
11591225
bump(f"textures_pal_{source}")
1160-
rows.append([str(rel), f"0x{off:x}", len(blob), sha[:12],
1161-
str(dest.relative_to(browse)), ""])
1226+
rows.append([str(rel), label, len(blob), sha[:12],
1227+
str(dest.relative_to(browse)), "", pk])
11621228
if i % 500 == 0:
11631229
log(f" {i}/{len(carriers)} "
11641230
f"({stats.get('textures_embedded_png', 0)} embedded PNGs so far)")
@@ -1167,12 +1233,13 @@ def bump(key: str, n: int = 1) -> None:
11671233
with open(browse / "embedded_textures.csv", "w", newline="") as fh:
11681234
w = _csv.writer(fh)
11691235
w.writerow(["carrier", "offset", "size", "sha1",
1170-
"written", "duplicate_of"])
1236+
"written", "duplicate_of", "packed"])
11711237
w.writerows(rows)
11721238
log(f"textures: {stats.get('textures_embedded_png', 0)} embedded PNGs "
11731239
f"({stats.get('textures_embedded_dup', 0)} duplicates skipped, "
1174-
f"{stats.get('textures_embedded_undecodable', 0)} undecodable) "
1175-
f"-> textures_png/_embedded/ + embedded_textures.csv")
1240+
f"{stats.get('textures_embedded_undecodable', 0)} undecodable; "
1241+
f"{stats.get('textures_embedded_arx', 0)} ARX-packed members "
1242+
f"decompressed) -> textures_png/_embedded/ + embedded_textures.csv")
11761243

11771244
if "audio" in kinds:
11781245
adir = browse / "audio"

‎cli.py‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,8 @@
3434
``browse`` converts the extracted dump into viewable/playable formats under
3535
``OUTDIR/browse/``: .xtx textures -> PNG (pure Python) plus a disc-wide
3636
sweep for XTX blobs embedded in effect/scene/UI containers (~3,000 more,
37-
written to textures_png/_embedded + embedded_textures.csv), .vds/.vdm
37+
ARX-packed scene members decompressed first; written to
38+
textures_png/_embedded + embedded_textures.csv), .vds/.vdm
3839
voice -> 48 kHz WAV (pure Python), battle voice/SE banks
3940
(yamamoto/snd/sed/*.bin) -> per-sample WAVs, .jpg copied + PS2ICON3D .res
4041
unpacked (icon.sys / .ico), .txt recoded to UTF-8 + string tables sniffed

‎docs/BROWSING.md‎

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -86,7 +86,7 @@ Each `browse` kind writes one subtree, preserving `dump/` paths:
8686
| Directory | Count | From | Open with |
8787
|---|---|---|---|
8888
| `textures_png/` | 1,269 PNGs | `.xtx` (+ sibling `.lex` palettes) | anything |
89-
| `textures_png/_embedded/` | 1,365 PNGs (+ CSV manifest) | XTX blobs inside `.esd`/`.esp`/`.a`/`.bin`/`.npr`/`.rbg`/`.bxx` | anything |
89+
| `textures_png/_embedded/` | 1,363 PNGs (+ CSV manifest) | XTX blobs inside `.esd`/`.esp`/`.a`/`.bin`/`.npr`/`.rbg`/`.bxx` (ARX-packed members decompressed) | anything |
9090
| `audio/` | 104 WAVs | `.vds`/`.vdm`/`.vda` streams | anything |
9191
| `soundbanks/` | per-bank WAVs + `smd_catalog.csv` | `.swd`/`.smd` | anything |
9292
| `text/` | 588 UTF-8 files + 6 sniffed string tables | Shift-JIS `.txt`, text-bearing `.dat`/`.info`/`.res`/… | your editor |
@@ -107,9 +107,12 @@ Highlights per subtree:
107107
(`title.npr`, `help.npr`, the `.rbg` room backdrops). Each PNG is named
108108
`<carrier>_<hex offset>.png`; `browse/embedded_textures.csv` maps every
109109
find (including blobs skipped as byte-identical duplicates of an
110-
already-decoded texture). The ~500 "undecodable" rows are all in `.a`
111-
archives — an in-engine XTX variant whose sub-image pointers are GS
112-
addresses, with pixel data streamed separately.
110+
already-decoded texture); `packed` = `arx` marks members that only
111+
exist after in-place ARX decompression (the `scene/cf*.a` archives).
112+
Those 603 are each scene's private copies of NPC, enemy, object and
113+
map textures — every one byte-identical to a standalone `.xtx`, so the
114+
CSV records them as duplicates and doubles as a per-scene provenance
115+
map (the rows an older build reported as "undecodable").
113116
* **`text/chain0/karakama/evtitem.dat.strings.txt`** — every event-item
114117
description; **`text/chain0/tanaka/CASINO.res.strings.txt`** — the casino
115118
dialogue; the `nisimori/*.info.strings.txt` dev configs keep their

‎docs/FORMATS.md‎

Lines changed: 57 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -78,16 +78,22 @@ Header size_comp = whole blob length incl. header; unk = 0.
7878
`scene/cf0210.a`, `cf0740.a`, `cf1800.a`, `cf3140.a` — 12 hair-palette
7979
carriers total (verified by disc-wide sweep; quarter-row sweep confirms
8080
no others).
81-
* **cf*.a re-frames the texture**: XTX magic + sub-header present but the
82-
entry layout differs and the canvas is stored with 4-byte inserts at a
83-
~2020-byte effective row stride, so only some 64-byte CLUT rows survive
81+
* **cf*.a members are ARX-compressed in place** (understood 2026-09-24;
82+
see "Scene archives" below). What earlier looked like a "re-framed"
83+
canvas with 4-byte inserts at a ~2020-byte stride is the ARX bit
84+
stream: the coder passes literal 32-bit words through verbatim with
85+
control words interleaved, so only *some* 64-byte CLUT rows survive
8486
contiguously (cf0210.a: 26/32 half-rows intact, 6 split). A row-level
8587
sweep alone therefore *partially* patches them — caught in the act via
8688
PINE RAM forensics: the running game builds the opening-sim KOS-MOS
8789
from cf0210.a, and its in-RAM CLUT showed exactly the 30 row-patched
88-
entries pink and 91 blue. Fix: anchored entry-level (aligned 4-byte
89-
value) replacement; safe because the hair ramp shares no exact RGBA
90-
word with any other tile in the canvas (verified 0 overlap).
90+
entries pink and 91 blue. The shipped fix — anchored entry-level
91+
(aligned 4-byte value) replacement — works because every hair-ramp
92+
word is a stream literal (none is one of the container's 30 LUT
93+
words) and shares no exact RGBA word with any other tile (verified 0
94+
overlap). The principled route is decompress → patch → `arx.compress`
95+
→ write back, which needs the `.a` TOC re-pointed when the container
96+
size changes; not built.
9197
* The engine also bakes scene lighting into CLUTs at load time (RAM
9298
copies differ from disc in RGB but not alpha), so RAM-vs-file palette
9399
comparisons must expect tinted variants.
@@ -182,6 +188,51 @@ as CT32 is more coherent, the block is drawn straight from the canvas
182188
finally renders the KOS-MOS hair band as hair instead of noise.
183189
Regions failing both readings stay garbled — the remaining static limit.
184190

191+
## Scene archives `.a` (+ `.fpk`, `.arc`) — ARX-packed members (`browse.py: iter_arx_containers`)
192+
193+
`scene/cf*.a` (112 files, one per field scene) is a flat bundle:
194+
195+
```
196+
u32 count; u32 offset[count] 16-byte-aligned; some entries point INTO a
197+
member (streaming pages), not all are starts
198+
record: 0x100 bytes of float data (placement/lighting; not parsed), then
199+
the member — usually an ARX container:
200+
"ARX\0" u32 usize u32 csize(includes this 16-byte header + 30-word LUT)
201+
u32 0; LUT[30]; bit stream -> arx.decompress(data[o:o+csize])
202+
member kinds (retail, 112 archives): 1,864 containers = lex 778,
203+
XTX 603, FPK 483. Uncompressed members exist too (JNT\0 joint tables,
204+
raw XTX sprites with their own 16x16 palette, raw 256x256 backdrops).
205+
(`.fpk`, `.arc` and a few `.bin`/`.npr` are whole-file ARX objects the
206+
TOC layer already decompresses; the 44 `mtnpack/*.arc` are FL00 wrappers
207+
of motion data, not Java — `carve_classes` finds nothing in them.)
208+
```
209+
210+
The ARX coder passes literal words through verbatim, so a packed XTX's
211+
**header survives inside the compressed stream**: `XTX\0`, a plausible
212+
total size and a sub-image table whose `file_addr` reads as a VIF DIRECT
213+
code (`0x5000xx01`) — that is what the raw sweep used to report as an
214+
"in-`.a` XTX variant with GS pointers, pixels streamed separately" (528
215+
undecodable hits on retail; all of them). There is no such variant. The
216+
sweep now decompresses every container first (`iter_arx_containers`:
217+
sane header + decompresses to exactly `usize`), skips raw hits that fall
218+
inside a container's span, and decodes the XTX members with the `lex`
219+
members packed beside them as their palette source (nearest member
220+
first — its material 0 seeds the base palette — the rest merged as
221+
companions, same as `kosmos_face.lex` on the standalone atlases).
222+
Result on retail: all 603 decode (0 undecodable) and every one is
223+
byte-identical to a standalone `char/`, `enemy/`, `obj/`, `map/` or
224+
`robo/` `.xtx` — each scene packs private copies of what it draws. The
225+
sweep therefore adds no new art, but `embedded_textures.csv` now maps
226+
every scene archive to the textures it carries (the "12 copies of the
227+
KOS-MOS palette" hunt in the Repack notes, answered disc-wide).
228+
229+
Dead ends worth not repeating: the "GS pointer" fields ARE consistent
230+
(`size` = (w*h*4+32)/16 qwords, addresses chain with no gaps) because
231+
they are the real header words; decoding the bytes after them as pixels
232+
finds coherent-looking art for the first ~60 KB (the stream's literal
233+
words are mostly pixel data) and then noise — the compressed stream is
234+
shorter than the texture, so the "pixels" run into the next member.
235+
185236
## LEX models — materials only (`browse.py: lex_materials`) — via xenotool
186237

187238
```

‎docs/UPDATES.md‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,43 @@
33
Newest first. Each entry lists what changed, why, and what you have to do
44
to benefit from it.
55

6+
## 2026-09-24 — scene-archive textures: the "undecodable variant" was ARX
7+
8+
The ~500 embedded XTX hits the sweep reported as *undecodable* (all in
9+
`scene/cf*.a`) were never a texture variant. Scene archives, `.fpk`
10+
packs and `.arc` bundles keep their members **ARX-compressed in place**,
11+
and the ARX coder passes literal words through verbatim — so a packed
12+
texture's *header* still reads as `XTX\0` plus sane sizes inside the
13+
compressed stream, while its pixels do not exist until decompression.
14+
15+
* `browse --kinds textures` now decompresses every in-file ARX container
16+
first (`browse.py: iter_arx_containers`; 1,864 inside the 112 retail
17+
scene archives: 603 XTX, 778 `lex` models, 483 FPK packs), decodes
18+
the XTX members, and palettes them with the `lex` models packed beside
19+
them (nearest member first). Raw hits inside a container's span are
20+
skipped, so the bogus rows are gone — and so are two "textures" the
21+
old sweep had decoded out of the middle of a compressed stream
22+
(`cf3021.a_208e4c`, `cf3070.a_61690c`: garbage).
23+
* Retail result: **1,363 embedded PNGs, 1,696 duplicates skipped, 0
24+
undecodable** (was 1,365 / 1,095 / 528). Nothing new to *look at*:
25+
all 603 packed textures are byte-identical copies of standalone
26+
`.xtx` you already have — the NPC skins (`char/`, 266), enemies
27+
(`enemy/`, 197), objects (`obj/`, 66), map atlases (`map/`, 65) and
28+
mechs (`robo/`, 9) each field scene packs for itself. What is new is
29+
the **provenance map**: `browse/embedded_textures.csv` now lists, per
30+
scene archive, every texture it carries and which standalone PNG it
31+
equals (`packed` = `arx`). That is the query the recolor tooling
32+
needed by hand ("which `cf*.a` carry KOS-MOS's palette?"), answered
33+
for every texture on the disc.
34+
* `docs/FORMATS.md` gets the `.a` archive layout (TOC + 0x100-byte
35+
record header + ARX member) and corrects the repack note: the
36+
"re-framed canvas with 4-byte inserts" that `pinkhair.py` works around
37+
is the ARX bit stream (the entry-level sweep works because the hair
38+
words are stream literals).
39+
40+
To pick this up: re-run `browse --kinds textures` on an existing dump
41+
(no re-extract). Pure-Python ARX adds ~2 minutes to the sweep.
42+
643
## 2026-08-21 — texture decode overhaul, embedded content sweep, BGM tempo fix
744

845
Full retest of the extractor against the retail USA ISO (two complete

0 commit comments

Comments
 (0)