From e45cd27ecebc602fd1b50daf5241448a292a0dc6 Mon Sep 17 00:00:00 2001 From: openfang Date: Fri, 31 Jul 2026 20:08:21 +0000 Subject: [PATCH 1/3] fix(runtime): LLM drivers had no HTTP timeout, so hangs made fallback unreachable MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit All four LLM driver HTTP clients (openai, anthropic, gemini, bedrock) were built with `reqwest::Client::builder()` and never set a timeout. reqwest applies NO timeout by default, so a provider that accepts the TCP connection and then never answers hangs the agent forever. The delay is not the worst part. A hang never returns `Err`, so `FallbackDriver` never fires — the fallback chain is unreachable precisely when it is needed. The mechanism that exists to survive a bad provider is disabled by the exact failure mode it was built for. Converting a hang into an error is what makes fallback work at all. Observed in production: an agent sat 15 minutes on a single request and made zero tool calls, while the same model answered a direct probe in seconds. Every other HTTP client in the repo already sets timeouts — web_search (15s), web_fetch (configurable), a2a (30s), provider_health (connect + overall), copilot token exchange, tool_runner (30s/15s). Only the LLM drivers, the ones that gate every agent turn, did not. Adds two constants in openfang-runtime: LLM_CONNECT_TIMEOUT_SECS = 15 (TCP + TLS only; unreachable => fail over fast) LLM_READ_TIMEOUT_SECS = 300 (stall detector) `read_timeout` is deliberate, not `timeout`. `timeout` caps total request duration and would kill long legitimate generations and long streams. `read_timeout` bounds the gap between successive bytes, so a slow-but-alive connection is fine while a dead one fails. 300s is kept under DEFAULT_AGENT_TURN_TIMEOUT_SECS (600) so the driver, not the turn supervisor, gives up first — the error stays attributable to the provider and a fallback still has time to run. CopilotDriver delegates completions to OpenAIDriver, so it inherits the fix. Co-Authored-By: Claude Opus 5 --- .../openfang-runtime/src/drivers/anthropic.rs | 8 ++++++ .../openfang-runtime/src/drivers/bedrock.rs | 8 ++++++ crates/openfang-runtime/src/drivers/gemini.rs | 8 ++++++ crates/openfang-runtime/src/drivers/openai.rs | 16 +++++++++++ crates/openfang-runtime/src/lib.rs | 28 +++++++++++++++++++ 5 files changed, 68 insertions(+) diff --git a/crates/openfang-runtime/src/drivers/anthropic.rs b/crates/openfang-runtime/src/drivers/anthropic.rs index 10a606e6a7..3fbcc40cdf 100644 --- a/crates/openfang-runtime/src/drivers/anthropic.rs +++ b/crates/openfang-runtime/src/drivers/anthropic.rs @@ -29,6 +29,14 @@ impl AnthropicDriver { base_url, client: reqwest::Client::builder() .user_agent(crate::USER_AGENT) + // reqwest applies no timeout by default: without these a provider + // that accepts the connection and never answers hangs the agent + // forever, and because a hang never returns Err the fallback chain + // can never fire. See crate::LLM_READ_TIMEOUT_SECS. + .connect_timeout(std::time::Duration::from_secs( + crate::LLM_CONNECT_TIMEOUT_SECS, + )) + .read_timeout(std::time::Duration::from_secs(crate::LLM_READ_TIMEOUT_SECS)) .build() .unwrap_or_default(), } diff --git a/crates/openfang-runtime/src/drivers/bedrock.rs b/crates/openfang-runtime/src/drivers/bedrock.rs index 4920e0b3c3..350ee59704 100644 --- a/crates/openfang-runtime/src/drivers/bedrock.rs +++ b/crates/openfang-runtime/src/drivers/bedrock.rs @@ -45,6 +45,14 @@ impl BedrockDriver { region: resolved_region, client: reqwest::Client::builder() .user_agent(crate::USER_AGENT) + // reqwest applies no timeout by default: without these a provider + // that accepts the connection and never answers hangs the agent + // forever, and because a hang never returns Err the fallback chain + // can never fire. See crate::LLM_READ_TIMEOUT_SECS. + .connect_timeout(std::time::Duration::from_secs( + crate::LLM_CONNECT_TIMEOUT_SECS, + )) + .read_timeout(std::time::Duration::from_secs(crate::LLM_READ_TIMEOUT_SECS)) .build() .unwrap_or_default(), }) diff --git a/crates/openfang-runtime/src/drivers/gemini.rs b/crates/openfang-runtime/src/drivers/gemini.rs index aee4e30c78..f32531df0e 100644 --- a/crates/openfang-runtime/src/drivers/gemini.rs +++ b/crates/openfang-runtime/src/drivers/gemini.rs @@ -34,6 +34,14 @@ impl GeminiDriver { base_url, client: reqwest::Client::builder() .user_agent(crate::USER_AGENT) + // reqwest applies no timeout by default: without these a provider + // that accepts the connection and never answers hangs the agent + // forever, and because a hang never returns Err the fallback chain + // can never fire. See crate::LLM_READ_TIMEOUT_SECS. + .connect_timeout(std::time::Duration::from_secs( + crate::LLM_CONNECT_TIMEOUT_SECS, + )) + .read_timeout(std::time::Duration::from_secs(crate::LLM_READ_TIMEOUT_SECS)) .build() .unwrap_or_default(), } diff --git a/crates/openfang-runtime/src/drivers/openai.rs b/crates/openfang-runtime/src/drivers/openai.rs index 73210f2757..7e9c88c8d8 100644 --- a/crates/openfang-runtime/src/drivers/openai.rs +++ b/crates/openfang-runtime/src/drivers/openai.rs @@ -34,6 +34,14 @@ impl OpenAIDriver { base_url, client: reqwest::Client::builder() .user_agent(crate::USER_AGENT) + // reqwest applies no timeout by default: without these a provider + // that accepts the connection and never answers hangs the agent + // forever, and because a hang never returns Err the fallback chain + // can never fire. See crate::LLM_READ_TIMEOUT_SECS. + .connect_timeout(std::time::Duration::from_secs( + crate::LLM_CONNECT_TIMEOUT_SECS, + )) + .read_timeout(std::time::Duration::from_secs(crate::LLM_READ_TIMEOUT_SECS)) .build() .unwrap_or_default(), extra_headers: Vec::new(), @@ -52,6 +60,14 @@ impl OpenAIDriver { base_url, client: reqwest::Client::builder() .user_agent(crate::USER_AGENT) + // reqwest applies no timeout by default: without these a provider + // that accepts the connection and never answers hangs the agent + // forever, and because a hang never returns Err the fallback chain + // can never fire. See crate::LLM_READ_TIMEOUT_SECS. + .connect_timeout(std::time::Duration::from_secs( + crate::LLM_CONNECT_TIMEOUT_SECS, + )) + .read_timeout(std::time::Duration::from_secs(crate::LLM_READ_TIMEOUT_SECS)) .build() .unwrap_or_default(), extra_headers: Vec::new(), diff --git a/crates/openfang-runtime/src/lib.rs b/crates/openfang-runtime/src/lib.rs index bde54ab199..eaec52a337 100644 --- a/crates/openfang-runtime/src/lib.rs +++ b/crates/openfang-runtime/src/lib.rs @@ -7,6 +7,34 @@ /// Some LLM providers (e.g. Moonshot, Qwen) reject requests without one. pub const USER_AGENT: &str = "openfang/0.3.48"; +/// Connect timeout for LLM provider requests, in seconds. +/// +/// Only covers TCP + TLS establishment, so it can be short: a provider that +/// cannot be reached should fail over immediately rather than stalling a turn. +pub const LLM_CONNECT_TIMEOUT_SECS: u64 = 15; + +/// Read timeout for LLM provider requests, in seconds. +/// +/// This is a *stall* detector, not a budget for the whole request: it bounds +/// the gap between successive bytes, so a long generation or a long stream is +/// fine as long as the connection keeps producing something. +/// +/// `reqwest` applies NO timeout by default. Without this, a provider that +/// accepts the connection and then never answers hangs the agent **forever** — +/// observed repeatedly against NVIDIA NIM, where an agent sat 15 minutes on a +/// single request and made zero tool calls while the same model answered a +/// direct probe in seconds. +/// +/// The critical consequence is not just the delay. A hang never returns `Err`, +/// so `FallbackDriver` never fires: the fallback chain is unreachable precisely +/// when it is needed. Converting a hang into an error is what makes fallback +/// work at all. +/// +/// Kept well under `DEFAULT_AGENT_TURN_TIMEOUT_SECS` (600) so the driver, not +/// the turn supervisor, is what gives up first — that way the error is +/// attributable to the provider and a fallback still has time to run. +pub const LLM_READ_TIMEOUT_SECS: u64 = 300; + pub mod a2a; pub mod agent_context; pub mod agent_loop; From e33f459f62651729e616813a6d39a8fc85d9864c Mon Sep 17 00:00:00 2001 From: openfang Date: Fri, 31 Jul 2026 20:10:41 +0000 Subject: [PATCH 2/3] fix(runtime): vertex + embedding clients also had no HTTP timeout MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Follow-up to the previous commit on this branch, which patched the five LLM driver clients built with `Client::builder()`. These two were missed because they construct the client with bare `reqwest::Client::new()` instead, so a search for `builder()` did not surface them — same bug, same consequence. `vertex.rs` is the more serious of the two: it is a full chat-completion driver, so with no read timeout a Vertex endpoint that accepts the connection and never answers hangs the agent forever. A hang never returns `Err`, so `FallbackDriver` never fires — the fallback chain is unreachable precisely when it is needed. `vertex.rs` was additionally missing `.user_agent(crate::USER_AGENT)`, which every other driver sets; the constant's doc comment notes that some providers reject requests that arrive without one. Added here too. `embedding.rs` gets a deliberately shorter read timeout via a new `EMBEDDING_READ_TIMEOUT_SECS` (60s). An embedding call has no generation phase and normally returns in well under a second, and failing it is cheap: `agent_loop` already logs "falling back to text search" and continues with keyword recall when `embed_one` returns `Err`. Waiting the full 300s would stall the turn before doing the same work the fallback would have done immediately. The two `Client::builder()` sites in `copilot.rs` were audited and left alone: they already set a total `.timeout()` (10s token exchange, 30s device flow) and are OAuth-only clients, not chat paths. Not compiled or tested locally (no build on this host); CI must verify. Co-Authored-By: Claude Opus 5 --- crates/openfang-runtime/src/drivers/vertex.rs | 13 ++++++++++++- crates/openfang-runtime/src/embedding.rs | 16 +++++++++++++++- crates/openfang-runtime/src/lib.rs | 15 +++++++++++++++ 3 files changed, 42 insertions(+), 2 deletions(-) diff --git a/crates/openfang-runtime/src/drivers/vertex.rs b/crates/openfang-runtime/src/drivers/vertex.rs index 9f04841634..817718323a 100644 --- a/crates/openfang-runtime/src/drivers/vertex.rs +++ b/crates/openfang-runtime/src/drivers/vertex.rs @@ -91,7 +91,18 @@ impl VertexAIDriver { project_id, region, token_cache: Arc::new(RwLock::new(TokenCache::new())), - client: reqwest::Client::new(), + client: reqwest::Client::builder() + .user_agent(crate::USER_AGENT) + // reqwest applies no timeout by default: without these a provider + // that accepts the connection and never answers hangs the agent + // forever, and because a hang never returns Err the fallback chain + // can never fire. See crate::LLM_READ_TIMEOUT_SECS. + .connect_timeout(std::time::Duration::from_secs( + crate::LLM_CONNECT_TIMEOUT_SECS, + )) + .read_timeout(std::time::Duration::from_secs(crate::LLM_READ_TIMEOUT_SECS)) + .build() + .unwrap_or_default(), } } diff --git a/crates/openfang-runtime/src/embedding.rs b/crates/openfang-runtime/src/embedding.rs index c3245d879c..7d8753ea29 100644 --- a/crates/openfang-runtime/src/embedding.rs +++ b/crates/openfang-runtime/src/embedding.rs @@ -96,7 +96,21 @@ impl OpenAIEmbeddingDriver { api_key: Zeroizing::new(config.api_key), base_url: config.base_url, model: config.model, - client: reqwest::Client::new(), + client: reqwest::Client::builder() + .user_agent(crate::USER_AGENT) + // reqwest applies no timeout by default: without these a provider + // that accepts the connection and never answers stalls the turn + // forever instead of returning Err, so the documented fallback to + // text search in agent_loop never runs. The read timeout is much + // shorter than the chat one — see crate::EMBEDDING_READ_TIMEOUT_SECS. + .connect_timeout(std::time::Duration::from_secs( + crate::LLM_CONNECT_TIMEOUT_SECS, + )) + .read_timeout(std::time::Duration::from_secs( + crate::EMBEDDING_READ_TIMEOUT_SECS, + )) + .build() + .unwrap_or_default(), dims, }) } diff --git a/crates/openfang-runtime/src/lib.rs b/crates/openfang-runtime/src/lib.rs index eaec52a337..552764550a 100644 --- a/crates/openfang-runtime/src/lib.rs +++ b/crates/openfang-runtime/src/lib.rs @@ -35,6 +35,21 @@ pub const LLM_CONNECT_TIMEOUT_SECS: u64 = 15; /// attributable to the provider and a fallback still has time to run. pub const LLM_READ_TIMEOUT_SECS: u64 = 300; +/// Read timeout for embedding provider requests, in seconds. +/// +/// Deliberately much shorter than `LLM_READ_TIMEOUT_SECS`: an embedding call +/// has no generation phase, returns a fixed-size vector, and normally completes +/// in well under a second, so a stall of a minute already means the provider is +/// not coming back. +/// +/// Failing fast is also cheap here, unlike a chat completion. Embeddings are +/// only used to *upgrade* memory recall to vector similarity search, and +/// `agent_loop` already logs "falling back to text search" and continues with +/// keyword recall when `embed_one` returns `Err`. Waiting the full 300s would +/// stall the turn before doing the same work the fallback would have done +/// immediately. +pub const EMBEDDING_READ_TIMEOUT_SECS: u64 = 60; + pub mod a2a; pub mod agent_context; pub mod agent_loop; From 3fbf564e2341325e57c16a08ec7468b0864a021d Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 1 Aug 2026 14:02:29 +0000 Subject: [PATCH 3/3] fix(collector): job sweeps surfaced the same 4 employers, not the same jobs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Across cycles 54-56 the remote-jobs beat delivered 25 roles from just four companies — Coinbase 11, Twilio 7, Stripe 5, Gusto 2 — out of 50 total. Dedup was not at fault: seen_jobs.json held 50 entries with 50 unique keys, zero repeats. Every one of those Coinbase roles was a genuinely different opening. The failure was upstream, in where the roles came from. Cycles 54-56 drew almost entirely from We Work Remotely. WWR's listing is ordered by recency and dominated by a handful of bulk posters, so an agent walking it top-down and taking ten spends its whole per-run budget inside three employers, then resumes from the same list next cycle. Cycles 53 and 57 — which used the HN "Who is hiring" thread, Greenhouse, Lever, Ashby and YC — returned 19 distinct companies with real comp bands. Same agent, same dedup, same day. The brief already forbade this. Hard filter 5 rejects "aggregator reposts with no original link", yet all thirty WWR-sourced roles recorded "apply via WWR" as the apply link and carried no comp data. The rule lost because the same section that stated it also listed three aggregators as sources under a heading that said to prefer origins — so the agent resolved the contradiction the cheap way. Three changes, all to the brief: - Split "Where to look" into originating boards (ATS, cite these) and discovery indexes (search these, never cite them), and require at least three distinct sources per run with no two consecutive runs opening on the same one. A single source producing every hit is the symptom to act on. - Add a Provenance section making the recorded URL the company's own posting or ATS entry, and drop any role whose original cannot be reached. Restates filter 5 where it is actually applied, since that is the rule most broken. - Cap each company at 2 roles per run. Ten roles at three companies is worth less than ten at ten, and the skipped openings are still there next cycle. Filter 5 is reworded to point at Provenance rather than repeat itself. Live copy on the PVC is already patched; this syncs the tracked copy. --- .../collector-briefs/remote-jobs.context.md | 46 ++++++++++++++++--- 1 file changed, 40 insertions(+), 6 deletions(-) diff --git a/deploy/797th/collector-briefs/remote-jobs.context.md b/deploy/797th/collector-briefs/remote-jobs.context.md index 759a1f57f6..2a2894bced 100644 --- a/deploy/797th/collector-briefs/remote-jobs.context.md +++ b/deploy/797th/collector-briefs/remote-jobs.context.md @@ -31,7 +31,7 @@ been shown. say so in `properties`. 5. **Real and directly applicable.** A link to the company's own posting or its ATS (Greenhouse, Lever, Ashby, Workable, Rippling). REJECT aggregator - reposts with no original link, staffing-agency listings, and "join our talent + listings cited as the apply link (see Provenance), staffing-agency listings, and "join our talent pool" evergreen pages. ## No duplicates — do this FIRST, every run @@ -58,18 +58,48 @@ the title was tweaked. Only a genuinely different opening counts. ## Where to look -Go to where postings originate, not where they are aggregated: +Postings ORIGINATE on a company's own board. Aggregators are a discovery +index, not a source. Both matter, but they play different roles — see +"Provenance" below. + +**Originating boards (preferred — these carry comp and a real apply link):** + +- company career pages on Greenhouse / Lever / Ashby / Workable / Rippling +- `"remote" "software engineer" site:boards.greenhouse.io` +- `"remote" "software engineer" site:jobs.lever.co` +- YC "Work at a Startup", Wellfound / AngelList remote roles + +**Discovery indexes (use to FIND roles, never to cite them):** - the monthly Hacker News **"Ask HN: Who is hiring?"** thread — filter to REMOTE - `weworkremotely.com`, `remoteok.com`, `remotive.com` — software category -- Wellfound / AngelList remote roles, YC "Work at a Startup" -- company career pages on Greenhouse / Lever / Ashby job boards -- `"remote" "software engineer" site:boards.greenhouse.io` -- `"fully remote" backend engineer hiring ` + +**Rotate your sources every run.** Use at least THREE distinct sources per +run, and do not begin two consecutive runs with the same source. If one +source is producing every role you find, you are walking a single list +top-down and will surface the same few bulk-posting employers forever — +stop and go somewhere else. Use the ACTUAL current year and month from the Current Date you are given. Never assume a year. +## Provenance — the apply link must be the real one + +For every role you report, the recorded `url` MUST be the company's own +posting or its ATS entry (Greenhouse, Lever, Ashby, Workable, Rippling), or +the company's careers page for that specific role. + +If you found a role on We Work Remotely, RemoteOK, Remotive, or any other +index, that is where you FOUND it — it is not the posting. Follow through to +the original and record that link. If you cannot reach an original posting, +DROP the role. Never record an aggregator URL as the apply link, and never +write "apply via WWR", "apply via RemoteOK", or similar. Hard filter 5 +already forbids this; it is restated here because it is the rule most often +broken. + +Cite the index in the report's `Sources` section if you used it — that is +what the Sources section is for. + ## Recording to the knowledge graph Use ONLY the configured vocabulary. The kernel REJECTS labels outside it and a @@ -132,6 +162,10 @@ cycle — a quiet cycle is a correct outcome. ## Non-negotiables +- **At most 2 roles per company per run.** One employer must never dominate + a cycle. On hitting 2 for a company, skip its remaining openings — they + will still be there next run, and a list of ten roles at three companies + is worth less than ten roles at ten companies. - **At most 10 new roles per run.** Count as you go. On hitting 10, or once roughly half your tool calls are spent, STOP COLLECTING and write the report. A run that exhausts its iterations delivers NOTHING.