Fix ADTOF on ROCm, authenticate HuggingFace downloads, rebuild the macOS variant - #31
Merged
Merged
Conversation
pyproject.toml.ROCM is a copy-over-the-top variant that only swaps the torch wheel index, so it is never touched during normal development and silently misses every dependency added to the CUDA pyproject. That is how adtof-pytorch came to be absent from it: on AMD hardware the ADTOF drum transcription pipeline had no package to import at all. Add adtof-pytorch to the ROCm variant — its own requirements are torch, librosa, pretty_midi and numpy, all already declared there and all platform-neutral, so nothing else needs to move. Then stop the class of bug rather than just this instance: tests/test_pyproject_variants.py diffs the two dependency lists by PEP 503 canonical name and by version specifier, in both directions, and fails on any difference not recorded in PLATFORM_ONLY with a reason. A third test keeps PLATFORM_ONLY itself honest, so a stale exemption cannot quietly re-open the hole. Removing the adtof line reproduces the original failure; restoring it passes. The guard reads only TOML, so it runs in the rocm-resolve job (before the slow resolve) as well as in the main test job. pyproject.toml.MAC is deliberately left out of the diff: it predates the web UI — still declaring dearpygui, with no fastapi or uvicorn — and needs a rewrite before a comparison against it would mean anything. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vb2foWspag8yfjz1hvxas2
The Roformer loader fetched checkpoints with urllib.request.urlretrieve,
which sends no credentials. huggingface.co answers an anonymous request
for a gated repo with HTTP 401, so loading roformer-jarredou-6stem — the
one Roformer spec hosted on the Hub — failed with an opaque "Download
failed: HTTP Error 401", with nothing to tell the user a token was what
was missing.
Add utils/hf_hub.py, which owns both halves of the problem:
get_hf_token() resolves the token once, in one place: the token saved
through the Synth tab's banner first, then whatever
huggingface_hub finds (HF_TOKEN, huggingface-cli login).
models/musicgen_loader.py had this logic inline and now
calls the shared version, so the two cannot diverge.
download_file() parses huggingface.co /resolve/ URLs into a repo id,
revision, filename and repo type, and fetches them with
hf_hub_download carrying that token. Anything else —
the GitHub-release and raw.githubusercontent checkpoints
the other two Roformer specs use — takes the plain path
unchanged, since there is nothing there to authenticate.
Both paths stay atomic: the file only appears at its destination once the
bytes are complete, so an interrupted download can't leave a truncated
file that the next load treats as a cache hit. Hub downloads stage beside
the destination rather than in the shared HF cache, so a gigabyte
checkpoint isn't stored twice, and a symlinked result (huggingface_hub
< 0.26 could satisfy local_dir that way) is materialised rather than
moved, so pruning the Hub cache can't dangle our copy.
A 401/403/gated failure now raises a message naming the repo's URL and
the two steps to fix it, and distinguishes "you have no token" from "your
token was rejected" — a different fix each. Unrelated failures (timeouts,
missing files) pass through untouched rather than being mislabeled as an
auth problem.
tests/test_hf_hub.py covers URL parsing (nested paths, single-segment
repo ids, encoded revisions, dataset/space prefixes, and a look-alike
host that must not receive our token), the routing decision in both
directions, partial-file cleanup, and the error text.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vb2foWspag8yfjz1hvxas2
pyproject.toml.MAC predated the web UI rewrite: it still declared
dearpygui and sounddevice, had no fastapi/uvicorn/httpx, no AceStep, no
Enhance/RVC/effects stack, floated torch at >=2.1.0, built a "gui"
package that no longer exists, and pointed its console script at
gui.app:main. Copying it over pyproject.toml produced an install that
could not run the application at all.
Rebuild it from the current CUDA pyproject so the dependency lists match
line for line, with three deliberate differences, each recorded in the
parity test's exception table:
- torch/torchaudio/torchvision keep the same 2.10.0 pins but come from
PyPI — the macOS arm64 wheels carry MPS, so no wheel index is needed
- nano-vllm is dropped: ACE-Step's own environment marker excludes it
on darwin/arm64 (its LM runs through MLX there), and its flash-attn
requirements are CUDA wheel URLs
- nvidia-cuda-runtime-cu12 goes with it — it existed only to provide
the CUDA runtime that flash-attn links against
pyworld and diffq publish no cp312 macOS wheels and build from sdist,
noted in the header and README (Xcode command line tools required).
Generalize tests/test_pyproject_variants.py from a CUDA/ROCm pair diff
to all three variants: every dependency must be declared everywhere,
PLATFORM_EXCEPTIONS records which variants may omit a package and why,
and exceptions are exact — a package declared where its entry says it
must be absent fails too, so a stale exemption cannot hide new drift.
The rocm-resolve CI job becomes a variant-resolve matrix over ROCM and
MAC; the darwin/arm64 graph resolves from a Linux runner because the
variant pins its target environment, and the full resolve was verified
here (AceStep resolves for darwin/arm64 with mlx/mlx-lm, adtof-pytorch
and diffq included; lock restored afterwards, as the committed uv.lock
stays cu130).
Also bring CLAUDE.md and README back to reality:
- BS-Roformer: 3 registered models, not 6 — roformer-kj-vocals and the
"+ 2 more" never existed in models/registry.py
- project tree: add the files the map had fallen behind on
(transcribe/voice API routers, transcribe/RVC pipelines,
sfx_renderer, music21_bridge, the four pitch-shift utils)
- macOS platform notes rewritten to match the new variant; the README
limitations section loses two claims that are no longer true
(ai-edge-litert now ships macOS arm64 wheels, and no code ever hid
mdx_extra_q from the UI) and gains the sdist-build note
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Vb2foWspag8yfjz1hvxas2
Both branches appended test files to the same pytest invocation in ci.yml; the resolution keeps all three additions. CLAUDE.md and README auto-merged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Vb2foWspag8yfjz1hvxas2
Keep the variant-resolve matrix job and parity-check step from this branch; take main's setup-uv bump to v10.0.1. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HxPcFzqTzXT4G86yLrTY71
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three related fixes, one commit each.
1.
adtof-pytorchmissing frompyproject.toml.ROCM(+ variant drift guard)The ROCm variant is copied over
pyproject.tomland never touched during normal development, so it silently missedadtof-pytorch— on AMD hardware the ADTOF drum transcription pipeline had no package to import. Added it (its own deps — torch, librosa, pretty_midi, numpy — are already declared and platform-neutral).To stop the class of bug,
tests/test_pyproject_variants.pydiffs the variants' dependency lists by PEP 503 canonical name and by version specifier, in both directions. Legitimate platform splits live in an exception table with reasons (nvidia-cuda-runtime-cu12,triton-rocm,nano-vllmon macOS), and the exceptions are exact — a stale exemption fails too. Removing theadtof-pytorchline reproduces the original failure. The guard runs in the resolve job (it only reads TOML, so it fires before the slow resolve) and in the test job.2. BS-Roformer downloads used unauthenticated
urllib→ HTTP 401models/roformer_loader.pyfetched checkpoints withurllib.request.urlretrieve, which sends no credentials — huggingface.co answers a gated repo with 401, soroformer-jarredou-6stem(the one Hub-hosted Roformer spec) failed with an opaque "Download failed".New
utils/hf_hub.py:get_hf_token()— one resolution order everywhere: the UI-saved token, thenHF_TOKEN/huggingface-cli login.models/musicgen_loader.pyhad this inline and now shares it.download_file()— parseshuggingface.co/.../resolve/...URLs (nested paths, encoded revisions, dataset/space prefixes; look-alike hosts are rejected so the token can't leak) and fetches viahf_hub_downloadwith the token. GitHub-release /raw.githubusercontent.comcheckpoints keep the plain path.Both paths are atomic (no truncated file can pose as a cache hit), Hub downloads stage beside the destination instead of duplicating a GB in the shared HF cache, and 401/403 now raises a message naming the repo URL and distinguishing "no token" from "token rejected". 19 tests in
tests/test_hf_hub.py.3.
pyproject.toml.MACrebuilt; CLAUDE.md/README brought back to realityThe macOS variant predated the web UI: it still declared
dearpygui, had nofastapi/uvicorn, no AceStep, floated torch at>=2.1.0, and built aguipackage that no longer exists — copying it in produced an install that couldn't run the app. It is now rebuilt from the current CUDA file, matching line for line except:nano-vllmdropped (ACE-Step's marker excludes it on darwin/arm64 — the LM runs through MLX; its flash-attn requirements are CUDA wheel URLs) andnvidia-cuda-runtime-cu12with itThe parity guard now covers all three variants, and the
rocm-resolveCI job becomes avariant-resolvematrix over ROCM + MAC (the darwin/arm64 graph resolves from Linux because the variant pins its target environment — verified locally, with AceStep resolving via mlx/mlx-lm; the committeduv.lockstays cu130).Docs: CLAUDE.md's BS-Roformer claims corrected (3 registered models, not 6 —
roformer-kj-vocalsnever existed in the registry), the project tree catches up on ~9 undocumented modules, and README's macOS limitations lose two no-longer-true claims (ai-edge-litertnow ships macOS arm64 wheels; nothing ever hidmdx_extra_qfrom the UI) and gain the sdist-build note forpyworld/diffq.Validation
tests/test_pyproject_variants.py+tests/test_hf_hub.py: 22 passed locally; drift guard verified to fail on the original bug and on stale exemptionsuv lockrun against both the ROCM probe and the full MAC variant; lock restored afterwardsci.ymlparses; both new test files wired into the CPU test jobmacOS resolution is CI-verified but untested on real hardware.
🤖 Generated with Claude Code
https://claude.ai/code/session_01Vb2foWspag8yfjz1hvxas2
Generated by Claude Code