Repository navigation
fix(collector): job sweeps surfaced the same 4 employers, not the same jobs - #21
Merged
Merged
Conversation
… unreachable All four LLM driver HTTP clients (openai, anthropic, gemini, bedrock) were built with `reqwest::Client::builder()` and never set a timeout. reqwest applies NO timeout by default, so a provider that accepts the TCP connection and then never answers hangs the agent forever. The delay is not the worst part. A hang never returns `Err`, so `FallbackDriver` never fires — the fallback chain is unreachable precisely when it is needed. The mechanism that exists to survive a bad provider is disabled by the exact failure mode it was built for. Converting a hang into an error is what makes fallback work at all. Observed in production: an agent sat 15 minutes on a single request and made zero tool calls, while the same model answered a direct probe in seconds. Every other HTTP client in the repo already sets timeouts — web_search (15s), web_fetch (configurable), a2a (30s), provider_health (connect + overall), copilot token exchange, tool_runner (30s/15s). Only the LLM drivers, the ones that gate every agent turn, did not. Adds two constants in openfang-runtime: LLM_CONNECT_TIMEOUT_SECS = 15 (TCP + TLS only; unreachable => fail over fast) LLM_READ_TIMEOUT_SECS = 300 (stall detector) `read_timeout` is deliberate, not `timeout`. `timeout` caps total request duration and would kill long legitimate generations and long streams. `read_timeout` bounds the gap between successive bytes, so a slow-but-alive connection is fine while a dead one fails. 300s is kept under DEFAULT_AGENT_TURN_TIMEOUT_SECS (600) so the driver, not the turn supervisor, gives up first — the error stays attributable to the provider and a fallback still has time to run. CopilotDriver delegates completions to OpenAIDriver, so it inherits the fix. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Follow-up to the previous commit on this branch, which patched the five LLM driver clients built with `Client::builder()`. These two were missed because they construct the client with bare `reqwest::Client::new()` instead, so a search for `builder()` did not surface them — same bug, same consequence. `vertex.rs` is the more serious of the two: it is a full chat-completion driver, so with no read timeout a Vertex endpoint that accepts the connection and never answers hangs the agent forever. A hang never returns `Err`, so `FallbackDriver` never fires — the fallback chain is unreachable precisely when it is needed. `vertex.rs` was additionally missing `.user_agent(crate::USER_AGENT)`, which every other driver sets; the constant's doc comment notes that some providers reject requests that arrive without one. Added here too. `embedding.rs` gets a deliberately shorter read timeout via a new `EMBEDDING_READ_TIMEOUT_SECS` (60s). An embedding call has no generation phase and normally returns in well under a second, and failing it is cheap: `agent_loop` already logs "falling back to text search" and continues with keyword recall when `embed_one` returns `Err`. Waiting the full 300s would stall the turn before doing the same work the fallback would have done immediately. The two `Client::builder()` sites in `copilot.rs` were audited and left alone: they already set a total `.timeout()` (10s token exchange, 30s device flow) and are OAuth-only clients, not chat paths. Not compiled or tested locally (no build on this host); CI must verify. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…e jobs Across cycles 54-56 the remote-jobs beat delivered 25 roles from just four companies — Coinbase 11, Twilio 7, Stripe 5, Gusto 2 — out of 50 total. Dedup was not at fault: seen_jobs.json held 50 entries with 50 unique keys, zero repeats. Every one of those Coinbase roles was a genuinely different opening. The failure was upstream, in where the roles came from. Cycles 54-56 drew almost entirely from We Work Remotely. WWR's listing is ordered by recency and dominated by a handful of bulk posters, so an agent walking it top-down and taking ten spends its whole per-run budget inside three employers, then resumes from the same list next cycle. Cycles 53 and 57 — which used the HN "Who is hiring" thread, Greenhouse, Lever, Ashby and YC — returned 19 distinct companies with real comp bands. Same agent, same dedup, same day. The brief already forbade this. Hard filter 5 rejects "aggregator reposts with no original link", yet all thirty WWR-sourced roles recorded "apply via WWR" as the apply link and carried no comp data. The rule lost because the same section that stated it also listed three aggregators as sources under a heading that said to prefer origins — so the agent resolved the contradiction the cheap way. Three changes, all to the brief: - Split "Where to look" into originating boards (ATS, cite these) and discovery indexes (search these, never cite them), and require at least three distinct sources per run with no two consecutive runs opening on the same one. A single source producing every hit is the symptom to act on. - Add a Provenance section making the recorded URL the company's own posting or ATS entry, and drop any role whose original cannot be reached. Restates filter 5 where it is actually applied, since that is the rule most broken. - Cap each company at 2 roles per run. Ten roles at three companies is worth less than ten at ten, and the skipped openings are still there next cycle. Filter 5 is reworded to point at Provenance rather than repeat itself. Live copy on the PVC is already patched; this syncs the tracked copy.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Across cycles 54–56 the
remote-jobsbeat delivered 25 of 50 roles from four companies — Coinbase 11, Twilio 7, Stripe 5, Gusto 2.Not a dedup bug
seen_jobs.jsonheld 50 entries with 50 unique keys, zero repeats. Every Coinbase role was a genuinely different opening. Dedup worked exactly as designed.Actual cause: source concentration
Cycles 54–56 drew almost entirely from We Work Remotely. WWR's listing is recency-ordered and dominated by a few bulk posters, so an agent walking it top-down and taking ten burns its whole per-run budget inside three employers — then resumes from the same list next cycle.
Same agent, same dedup, same day.
The rule already existed
Hard filter 5 rejects "aggregator reposts with no original link". All thirty WWR-sourced roles recorded "apply via WWR" as the apply link. The rule lost because the same section that stated it also listed three aggregators as sources under a heading saying to prefer origins — so the agent resolved the contradiction the cheap way.
Changes
Deployment
The live copy on the PVC is already patched and in effect —
context.mdis re-read every turn, so no rebuild is needed. This PR syncs the tracked copy so it doesn't drift.Not yet verified
The per-company cap and provenance rules take effect on the next sweep (22:30 UTC). Neither has been observed in a real run yet.
🤖 Generated with Claude Code