Send only error spans to Honeycomb behind a toggle and stop deploying Refinery

[stopachka]

Prod sends every span to a Refinery box (45 to 90 GB a day), which keeps traces with an error plus a small sample and forwards about 0.5 GB a day to Honeycomb. We mostly use Honeycomb for error alerts now. Everything else we read from the span logs that vector ships to S3. The box, its load balancer and the traffic cost about $79 a month.

This lets prod send spans to Honeycomb directly and only send the ones with an error status:

• A new honeycomb-error-spans-only? toggle, off by default. make-honeycomb-exporter wraps the OTLP exporter, and when the toggle is on it drops every span whose status is not ERROR. That is the status add-exception! sets, so silenced exceptions are not sent. With the toggle off every span is exported as before, so self-hosted setups with a Honeycomb key see no change.
• docker-compose.yml no longer sets HONEYCOMB_ENDPOINT, so the server uses the default https://api.honeycomb.io:443. Refinery forwarded spans with the key the server already sends in x-honeycomb-team, so the same key works.
• The publish-refinery-eb job and its change detection are removed. server/refinery stays in the repo for anyone who wants to run it.

The logging exporter is untouched. All spans still go to stdout and S3.

Startup: the toggle is read on every export batch, not when the tracer is built, because tracer/init runs before flags load. Before flags load it reads false. flags-impl/init fetches flags in the foreground before the HTTP server starts, so the only unfiltered spans are the few from the invalidator and WAL starting up. instant.flags does not depend on the tracer, and the tracer already required it.

From a sample of Sep 16 logs, error spans are about 47 a second (4.1 M a day). Honeycomb already receives these today through Refinery's keep-errors rule. 88% of them are permission-evaluation-failed on transacts, which shows up twice: once on permissioned-transaction/transact! and once on uncaught-exception.

What changes in Honeycomb with the toggle on: error events arrive as single spans instead of whole traces, and the sampled non-error traffic stops. The e2e invalidator tracking spans are no longer sent there, they are still in the logs.

Rollout:

1. In the config app, add a toggles entity with setting = honeycomb-error-spans-only? and toggled = true. Do this before deploying. Without it prod would send every span (about 1,200 a second) straight to Honeycomb.
2. Merge and deploy.
3. Check that errors still arrive in Honeycomb and that traffic on eb-refinery-elb drops to zero.
4. Terminate the refinery Elastic Beanstalk environment.

Release bundles built from older commits still point at Refinery. After step 4 a deploy of one of those would log export failures and send nothing to Honeycomb. Nothing else breaks.

The test covers an ok span, an error span and a silenced exception, with the toggle off and on.

🤖 Generated with Claude Code

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Advanced

Run ID: 71101800-df50-4289-8904-b3abc960e4af

📥 Commits

Reviewing files that changed from the base of the PR and between c024ddd and 8ab1234.

📒 Files selected for processing (3)

• server/src/instant/flags.clj
• server/src/instant/util/tracer.clj
• server/test/instant/util/tracer_test.clj

Included review availability: Your plan provides up to 10 included reviews per hour; 8 remain after this review.

---

📝 Walkthrough

Walkthrough

The change removes Refinery CI deployment and endpoint configuration. It adds a toggle-controlled Honeycomb exporter that can forward all spans or only error spans, with tests for both modes.

Changes

Refinery removal

Layer / File(s)	Summary
Remove Refinery deployment and endpoint configuration .github/workflows/clojure.yml, server/docker-compose.yml	CI no longer detects or publishes Refinery changes. The web service no longer defines the Refinery Honeycomb endpoint.

Configurable Honeycomb tracing

Layer / File(s)	Summary
Configure Honeycomb span filtering server/src/instant/flags.clj, server/src/instant/util/tracer.clj, server/test/instant/util/tracer_test.clj	The new toggle defaults to exporting all spans. When enabled, the Honeycomb exporter forwards only error spans. Tests cover both toggle states and span categories.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant BatchSpanProcessor
  participant HoneycombExporter
  participant OTLPExporter
  BatchSpanProcessor->>HoneycombExporter: export span batch
  HoneycombExporter->>HoneycombExporter: check honeycomb-error-spans-only?
  HoneycombExporter->>OTLPExporter: export selected spans
  OTLPExporter-->>BatchSpanProcessor: return export result

Loading

Suggested reviewers: dwwoelfel

Merge Risk: ⚪ Minimal · up to 8ab12

The configurable Honeycomb export behavior has no identified merge-blocking issue.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description check	✅ Passed	The description clearly explains the direct Honeycomb export, error-span toggle, Refinery deployment removal, rollout steps, and compatibility considerations.
Title check	✅ Passed	The title clearly summarizes the two primary changes: exporting only error spans to Honeycomb behind a toggle and stopping Refinery deployment.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}