What is wanted
evalgate list should ask the provider registry what is registered instead of printing a hardcoded string.
The problem
src/cli/index.ts, cmdList (lines 236-242):
const scorers = defaultScorerRegistry().list().sort();
console.log("Registered scorers:");
for (const s of scorers) console.log(` - ${s}`);
console.log("\nRegistered providers: mock, openai, anthropic, groq, openrouter");
The scorer half is derived from the registry. The provider half is a string literal. It happens to be correct right now (it matches the five register calls in defaultRegistry, src/providers/registry.ts lines 54-60), but nothing keeps it correct, and it can never reflect a provider registered at runtime, which is exactly what registryFor does for --degrade (line 60).
The reason it was written this way is that ScorerRegistry has a list() method (src/scorers/registry.ts lines 98-101) and ProviderRegistry does not. ProviderRegistry has register, has, and get, and already reads [...this.factories.keys()] internally at line 37 to build its unknown-provider error message.
Steps
- Add a
list(): string[] method to ProviderRegistry in src/providers/registry.ts, returning [...this.factories.keys()]. Mirror the docstring style of ScorerRegistry.list().
- Use it in
cmdList in src/cli/index.ts so providers print in the same bulleted, sorted form as scorers. cmdList currently takes no arguments; passing the registry built by registryFor(args) would also make evalgate list --degrade honest, though a plain defaultRegistry(process.env) is acceptable if you would rather keep the signature.
- Optionally simplify line 37 of
src/providers/registry.ts to reuse the new list().
- Add a small test, for example in
tests/providers.test.ts, asserting defaultRegistry({}).list() contains all five built-ins and that a register("custom", ...) call shows up.
Self-contained, under an hour. Comment below to claim it; I usually reply within a day.
What is wanted
evalgate listshould ask the provider registry what is registered instead of printing a hardcoded string.The problem
src/cli/index.ts,cmdList(lines 236-242):The scorer half is derived from the registry. The provider half is a string literal. It happens to be correct right now (it matches the five
registercalls indefaultRegistry,src/providers/registry.tslines 54-60), but nothing keeps it correct, and it can never reflect a provider registered at runtime, which is exactly whatregistryFordoes for--degrade(line 60).The reason it was written this way is that
ScorerRegistryhas alist()method (src/scorers/registry.tslines 98-101) andProviderRegistrydoes not.ProviderRegistryhasregister,has, andget, and already reads[...this.factories.keys()]internally at line 37 to build its unknown-provider error message.Steps
list(): string[]method toProviderRegistryinsrc/providers/registry.ts, returning[...this.factories.keys()]. Mirror the docstring style ofScorerRegistry.list().cmdListinsrc/cli/index.tsso providers print in the same bulleted, sorted form as scorers.cmdListcurrently takes no arguments; passing the registry built byregistryFor(args)would also makeevalgate list --degradehonest, though a plaindefaultRegistry(process.env)is acceptable if you would rather keep the signature.src/providers/registry.tsto reuse the newlist().tests/providers.test.ts, assertingdefaultRegistry({}).list()contains all five built-ins and that aregister("custom", ...)call shows up.Self-contained, under an hour. Comment below to claim it; I usually reply within a day.