Skip to content

rocm serve --engine lemonade: interactive variant selection for owner/repo #82

Description

@rominf

Motivation

rocm serve --engine lemonade <owner/repo:variant> serves a Hugging Face GGUF under its canonical name. Today a variant is required — a bare owner/repo, or a partial/ambiguous variant like :Q4 that matches several quantizations, is rejected with an actionable error rather than guessed (to avoid silently serving the wrong quantization). See PR #80.

A nicer UX would be to enumerate the available variants and let the user pick when the variant is missing or ambiguous, instead of erroring.

Proposed approach

  • Enumerate a repo's GGUF variants without downloading (Lemonade exposes a variants endpoint used by its own pull flow).
  • In the foreground/interactive serve path (which has a TTY, during the serve-plan phase in apps/rocm), present the variants and let the user choose, then proceed with the chosen owner/repo:variant.
  • Keep the current non-interactive behavior as the fallback: with no TTY (managed/background serves, scripts, CI), list the available variants in the error message so the user can specify one.

Notes / constraints

  • Interactive selection only applies to the foreground path; managed/background serves must keep the clear error.
  • Belongs in the apps/rocm foreground layer (TTY + serve plan), not the engine's detached serve_http.
  • Invariant to preserve: never silently pick a variant.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions