[agentic-token-optimizer] Optimize Daily Agentic AI Research Digest — reduce 8× AIC variance via fetch caps and content limits #229

Target Workflow

Daily Agentic AI Research Digest (daily-agentic-research.md)

Why selected: Third-highest AIC workflow in the repository (981 AIC / 7 runs over the analysis window). While recently analyzed, fresh run data reveals continued extreme AIC volatility (8× variance) driven by unconstrained web-fetch behavior — a structurally different problem from prior optimization passes. The AIC monitoring family workflows were excluded from consideration as higher priorities due to 7-day recency.

Analysis Period

7 runs from 2026-06-26 to 2026-07-02 (all successful, no failures).

Spend Profile

Key Finding: AIC Cost Is Turn-Linear

AIC tracks nearly linearly with turn count at ~9.45 AIC/turn. The 8× AIC variance (39 to 311) is driven entirely by turn count variance (4 to ~20), not model choice or prompt size. The agent consumes ~27,000 tokens/turn regardless of run — meaning every extra source fetch adds ~10 AIC.

The two most expensive runs (Jun 27–28, combined 544 AIC = 55% of weekly total) likely triggered 3+ source fetches including the broad arxiv search URL, which returns large result pages.

Ranked Recommendations

1. Harden the source-fetch limit and early-exit condition — estimated ~55 AIC/run savings

Current prompt (§ Browsing Instructions):

Browse 2–3 of these sources (don't fetch all if you find a strong candidate early).

Problem: "2–3" is permissive, and "if you find a strong candidate" is vague. On Jun 27 and Jun 28 the agent consumed 17–20 estimated turns (234–311 AIC). Capping at 2 sources and making the early-exit binary would eliminate these outlier runs.

Proposed change:

Fetch at most 2 sources. After each fetch, stop immediately if you find **any** paper or announcement published within the last 3 days — do not fetch a second source in that case.

Evidence: The cheapest run (Jun 30, 4 turns, 39 AIC) terminated early. The two most expensive (combined 544 AIC) did not. A hard-stop rule — not a suggestion — removes the ambiguity that causes overrun.

Estimated savings: Eliminating the two outlier runs and trimming 1–2 turns from mid-range runs projects to ~50–60 AIC/run reduction (36–43%).

2. Cap per-page content consumption — estimated ~18 AIC/run savings

Problem: At ~27,000 tokens/turn, the agent is consuming full pages including boilerplate, navigation chrome, and many result entries. The arxiv search URL alone can return 10+ full abstracts.

Proposed addition to §Browsing Instructions:

After fetching each page, extract only the first 6,000 characters. Do not scroll or paginate.

Evidence: The low-AIC run (Jun 30, 110,382 tokens across 4 turns = 27,596 tokens/turn) used the same pages. A 6,000-character cap per fetch would roughly halve per-turn token consumption, saving ~2 equivalent turns of AIC per run.

References:

• §28319198943 — highest-AIC run, agent artifact 1.84 MB vs 0.80 MB for recent run
• §28438420177 — most efficient run, 4 turns / 110,382 tokens

3. Replace broad arxiv search URL with max-results API endpoint — estimated ~10 AIC/run savings (when arxiv is fetched)

Current: `(arxiv.org/redacted)

This returns a full HTML search page with 10 results and full abstracts. The export API (export.arxiv.org — already allowlisted) returns structured Atom XML with a configurable result cap.

Proposed replacement:

(export.arxiv.org/redacted)

Evidence: export.arxiv.org is already in the network allowlist. Limiting to 3 results removes 70% of arxiv page content on fetches where this source is used. Conservative savings of ~1 turn equivalent (~10 AIC) per run where arxiv is visited.

Structural Optimization — Inline Sub-Agent for Browse-and-Extract

The workflow has no existing ## agent: blocks and has 4 major ## sections. The §Browsing Instructions section is a strong sub-agent candidate.

Candidate: browse-sources sub-agent

Why a smaller model fits: The browsing phase is entirely extractive — the agent matches paper titles against a date filter and extracts structured metadata. No judgment or synthesis is needed. A smaller model can identify "this paper is from July 1, here is its title and abstract" as reliably as the main model.

Proposed invocation change: Add a new ## agent: browse-sources block before ## Selection Criteria:

## agent: browse-sources

Fetch at most 2 of the following sources. For each page, extract all papers or announcements published in the last 7 days. Return a JSON array of objects with keys: title, url, date, one_sentence_summary. Stop after the first source if you find any item from the last 3 days.

Sources (priority order):
1. (huggingface.co/redacted)
2. (openai.com/redacted)

Respond with only the JSON array — no prose.

The main agent then receives the extracted candidates and applies the §Selection Criteria + §Output steps without reading raw HTML. This shifts ~60–70% of turn costs to a cheaper model.

Estimated savings: 30–50 AIC/run, depending on model cost differential. Conservative: ~30 AIC/run.

Caveats

• Token estimates for runs 11–12 (Jun 27–28) are derived from the consistent AIC-per-token ratio (0.351) across 5 measured runs; actual figures may differ.
• The arxiv export API returns Atom XML rather than HTML; the agent prompt may need to be updated to mention this format if it causes parsing confusion.
• The sub-agent recommendation depends on the agentic workflow framework's inline sub-agent support and the cost tier of the smaller model available.
• Recommendations 1–3 are independent and safe to apply individually.

Generated by Agentic Workflow AIC Usage Optimizer · 345.4 AIC · ⊞ 21.6K · ◷

• expires on Jul 9, 2026, 3:18 PM UTC