Investigate a structlog-native output path that skips the stdlib bridge

Every structlog call in lite-bootstrap goes structlog.stdlib.BoundLogger → lite-bootstrap's processors → JSONRenderer(orjson) → stdlib Logger.info(rendered) → MemoryHandler → StreamHandler (lite_bootstrap/instruments/logging_instrument.py, lite_bootstrap/instruments/logging_factory.py:109-133). The stdlib half is the main structural cost.

Measured on 2026-10-24 at 705d8a3 on Linux with 2 CPUs (CPython 3.14.9, output to /dev/null, 50000 calls per round), against a structlog-native configuration that is not lite-bootstrap code (make_filtering_bound_logger, BytesLoggerFactory, orjson bytes):

call	lite-bootstrap chain	structlog-native
`log.info("did a thing", a=1)`	9.5 µs	2.6 µs
`log.debug(...)` filtered at INFO	1.1 µs	0.2 µs

Roughly a third of a lite-bootstrap call is its processors, half is the stdlib bridge (Logger.makeRecord alone is 1.4 µs, Logger.info into a handler 3.8-4.1 µs), and the rest is structlog's dispatch. The MemoryHandler buys nothing per call: MemoryHandler.flush hands each buffered record to StreamHandler.emit, which writes and flushes each record on its own. The tables and scripts are to be published with #335.

The bridge is not dead weight. A native path loses what it provides today:

Sentry error events and breadcrumbs from structlog calls. LoggingIntegration only sees stdlib records, and enrich_sentry_event_from_structlog_log depends on that (ADR-0006).
logging_record_filters, which attach stdlib logging.Filters per logger.
Per-logger stdlib levels, and anything else a user hangs on stdlib handlers.

This is a question, not a plan: is a native path worth offering, for whom, and how does Sentry capture work on it?

Work

Decide whether to pursue it. If yes, write the design first, in this issue or an ADR:
- who opts in (a config flag, or only services without Sentry), and what happens when Sentry is configured too;
- how Sentry capture would work without the stdlib record, for example a structlog processor that calls sentry_sdk.capture_event or adds breadcrumbs, and what that costs per call;
- what logging_record_filters and per-logger levels become.
Measure the chosen design inside lite-bootstrap, with Sentry on, before any PR.

Verify: a decision recorded here, and if the answer is yes, a follow-up issue with the design and its measured cost.

Would be breaking only if it became the default.

Findings: structlog-native output path, measured inside lite-bootstrap

I prototyped a native path that runs lite-bootstrap's own processors and captures to Sentry from a structlog processor, and measured it against the current chain with and without Sentry. I also measured the intermediate step from the issue: keep the stdlib bridge and drop MemoryHandler.

Setup

Commit 83877be (origin/main), throwaway worktree, nothing in the repo changed.
Apple M2, 8 cores, macOS (Darwin 25.6.0), CPython 3.14.7.
structlog 26.1.0, sentry-sdk 2.73.0, orjson 3.13.0, uv sync --all-extras.
Each scenario runs in its own process, because sentry_sdk.init() patches Logger.callHandlers for good.
Config is FreeConfig defaults, built by the real LoggingInstrument and SentryInstrument. With Sentry: a dummy DSN and a transport that serializes each envelope and drops it, so serialization CPU is counted and there is no network.
Output goes to /dev/null: sys.stdout is rebound before bootstrap for the bridge, and the native path writes to a /dev/null binary file.
timeit.Timer with a warmup of a tenth of the round. info: 15 rounds of 20,000 calls. Filtered debug: 15 rounds of 200,000. error and exception: 9 rounds of 2,000. The table reports the median per call. Two full runs agreed within 4%; the numbers below are the second run.

Results (µs per call, median)

scenario	`info("did a thing", a=1)`	`debug` filtered at INFO	`error` (Sentry event)	`exception` (Sentry event)
bridge (today)	8.64	1.00	9.4	21.6
bridge + Sentry	14.29	1.04	2368	577
bridge without `MemoryHandler`	8.92	1.02	9.0	20.9
bridge without `MemoryHandler` + Sentry	14.80	1.04	2326	561
native	3.96	0.16	4.0	15.0
native + Sentry processor	7.74	0.16	1544	537
native + Sentry processor, breadcrumbs off	4.20	0.16	1320	356

Without Sentry, the native path cuts an info line from 8.6 to 4.0 µs and a filtered debug call from 1.0 to 0.16 µs. With Sentry, an info line goes from 14.3 to 7.7 µs.

The 4.0 µs is higher than the 2.6 µs in the issue because this run keeps lite-bootstrap's processors (ISO timestamper, tracer_injection, StackInfoRenderer, format_exc_info, UnicodeDecoder). Different machines, too.

On the native path, Sentry costs 3.8 µs per info line. The processor with breadcrumbs off costs 0.24 µs, so breadcrumbs are about 3.5 µs. sentry_sdk.add_breadcrumb alone measures 0.9 µs. Of the rest, fnmatch against sentry-sdk's three ignored logger names is 0.9 µs and can be cached per logger name. The remainder is building the data dict and the timestamp. On the bridge, Sentry costs 5.7 µs per info line, through the patched callHandlers and BreadcrumbHandler.

The event columns measure Sentry building and serializing an event. The logging path is a rounding error there. The plain error row is large because attach_stacktrace=True walks the current stack with local variables, and the benchmark stack has big locals. The absolute number depends on the caller's stack. The native event is cheaper mostly because the attached stack has fewer frames (6 against 9: the stdlib logging frames are gone).

Sentry capture on the native path

The prototype is two processors. stash_exc_info runs before format_exc_info and keeps the exception tuple. SentryCapture runs after logging_extra_processors and before the renderer. Both are built only when Sentry is configured.

def stash_exc_info(_, __, event_dict):
    exc_info = event_dict.get("exc_info")
    if exc_info:
        if isinstance(exc_info, BaseException):
            exc_info = (type(exc_info), exc_info, exc_info.__traceback__)
        elif not isinstance(exc_info, tuple):
            exc_info = sys.exc_info()
        if exc_info[0] is not None:
            event_dict["_sentry_exc_info"] = exc_info
    return event_dict


@dataclasses.dataclass(slots=True, frozen=True)
class SentryCapture:
    breadcrumb_level: int | None = logging.INFO
    event_level: int | None = logging.ERROR

    def __call__(self, _, __, event_dict):
        exc_info = event_dict.pop("_sentry_exc_info", None)
        level_name = event_dict["level"]
        levelno = _NAME_TO_LEVEL.get(level_name, logging.NOTSET)
        wants_event = self.event_level is not None and levelno >= self.event_level
        wants_crumb = self.breadcrumb_level is not None and levelno >= self.breadcrumb_level
        if not (wants_event or wants_crumb):
            return event_dict
        logger_name = event_dict.get("logger") or ""
        if any(fnmatch.fnmatch(logger_name, ignored) for ignored in sentry_logging._IGNORED_LOGGERS):
            return event_dict
        message = event_dict.get("event")
        extra = None
        if wants_event and not event_dict.get("skip_sentry"):
            client = sentry_sdk.get_client()
            if client.is_active():
                extra = {k: v for k, v in event_dict.items() if k not in _SENTRY_META_KEYS}
                if exc_info is not None:
                    event, hint = event_from_exception(
                        exc_info, client_options=client.options, mechanism={"type": "logging", "handled": True}
                    )
                else:
                    event, hint = {}, {}
                event["level"] = "fatal" if level_name == "critical" else level_name
                event["logger"] = logger_name
                event["logentry"] = {"message": message, "formatted": message, "params": ()}
                if extra:
                    event["contexts"] = {"structlog": extra}
                hint["structlog_event_dict"] = event_dict
                sentry_sdk.capture_event(event, hint=hint)
        if wants_crumb:
            if extra is None:
                extra = {k: v for k, v in event_dict.items() if k not in _SENTRY_META_KEYS}
            sentry_sdk.add_breadcrumb({
                "type": "log",
                "level": "fatal" if level_name == "critical" else level_name,
                "category": logger_name,
                "message": message,
                "timestamp": datetime.datetime.now(datetime.UTC),
                "data": extra,
            })
        return event_dict

_SENTRY_META_KEYS is STRUCTLOG_META_KEYS | {"_sentry_exc_info"}. The sentry-sdk APIs used are sentry_sdk.capture_event, sentry_sdk.add_breadcrumb, sentry_sdk.get_client and sentry_sdk.utils.event_from_exception, checked against 2.73.0. _IGNORED_LOGGERS is private to sentry-sdk; it is read so ignore_logger() keeps working.

I compared captured events from the same four calls on both paths (info, error, exception in an except, error with skip_sentry=True):

field	bridge today	native prototype
`level`, `logger`	`error`, `app.module`	same
`logentry.formatted`	the message, after `enrich_sentry_event_from_structlog_log` rewrites it	the message
`logentry.message`	the whole rendered line, JSON included (the seam only rewrites `formatted`)	the message
`contexts.structlog`	user keys, from parsing the rendered line	same keys, read from the event dict
exception from `log.exception()`	`ValueError: bad value`, mechanism `logging`, handled	same
stack on a plain `error`	attached by `attach_stacktrace`, 9 frames, the last four stdlib `logging`	attached, 6 frames, the last one the lite-bootstrap processor module
`skip_sentry=True`	no event, breadcrumb still added	same
breadcrumb `message`	the whole rendered line	the message
breadcrumb `data`	`{}`	user keys
`hint`	`log_record` (a `LogRecord`)	`structlog_event_dict`

enrich_sentry_event_from_structlog_log stays in before_send and does nothing for native events, because their formatted does not start with {. The native path does not need the ADR-0006 seam for structlog calls; foreign stdlib loggers still go through it.

Differences a user could notice:

Breadcrumbs carry the message and the user keys instead of a JSON line. This reads better in Sentry, but a saved search or alert that matches on the JSON text would stop matching.
A user before_send that reads hint["log_record"] gets nothing for structlog events.
logger is None for structlog.get_logger() called without a name. BytesLoggerFactory passes the first positional argument as the name and does not guess the caller's module. The name attribute on BytesLogger is new in structlog 26.1; the declared floor is 21.3, so the real change needs its own small logger class or a higher floor for this path.

Foreign stdlib loggers are unchanged on the native path: they still go through the root StreamHandler and LoggingIntegration. Structlog calls never reach stdlib, so nothing is captured twice.

`logging_record_filters` and per-logger levels

logging_record_filters keeps working for what the docs describe, filtering a third-party library's stdlib records, because foreign loggers stay on stdlib. A filter keyed on the name of a structlog logger would stop seeing structlog calls. On the native path, SentryCapture runs after logging_extra_processors, so an extra processor can rewrite level or raise DropEvent before Sentry sees the event. That is the native equivalent. The "too late to change what Sentry captured" rule in CONTEXT.md does not hold on this path, and the docs would need to say so. I would document this rather than translate filters into processors.

Per-logger levels are not available on the native path. make_filtering_bound_logger takes one global level, which is where the 0.16 µs filtered debug comes from. A per-name level map would be a processor raising DropEvent, and it would bring back most of the cost of building a filtered event.

This rules out two bootstrappers as they stand:

FastStreamBootstrapper calls structlog.get_logger("faststream").setLevel(faststream_log_level). FilteringBoundLogger has no setLevel.
LitestarLoggingInstrument hands wrapper_class=structlog.stdlib.BoundLogger and the stdlib factory to Litestar's StructlogPlugin.

Per-logger levels on the bridge are weaker than they look today, too: MemoryLoggerFactory.__call__ runs logger.setLevel(logging_log_level) every time a structlog proxy binds, so a level set on that stdlib logger beforehand is overwritten. I checked: logging.getLogger("y").setLevel(DEBUG) before the first structlog.get_logger("y").debug(...) leaves the level at 20 and drops the line.

Dropping `MemoryHandler` (the intermediate step)

Writing through a StreamHandler directly measures the same as today: 8.92 against 8.64 µs per info line, and 14.80 against 14.29 with Sentry, both inside run-to-run noise. The issue is right that MemoryHandler buys nothing per call.

It is a behaviour change, though, and the change is a fix:

With the defaults (logging_buffer_capacity=10, logging_flush_level=ERROR), structlog INFO lines wait in the buffer until there are 10 or an ERROR arrives, while foreign stdlib lines are written at once. structlog.info("1"); logging.getLogger("lib").warning("2"); structlog.info("3") prints 2, 1, 3.
On a hard exit (os._exit, SIGKILL, OOM kill) the buffered lines are lost. The same three calls followed by os._exit(0) print only line 2. A normal exit flushes them through logging.shutdown().
logging_buffer_capacity and logging_flush_level would do nothing and need a deprecation cycle. Many bootstrap tests already set logging_buffer_capacity=0.

This is worth a small separate change. The case for it is ordering and crash safety; it saves no time.

Two bugs found while measuring

Both are in MemoryLoggerFactory on the current path and worth their own issues:

Every structlog.get_logger(name) proxy adds another MemoryHandler to the same stdlib logger on first use. Two proxies for "x" print the second one's lines twice. Calling structlog.get_logger() inside a function that runs three times prints the third call three times, and the handlers pile up for the life of the process.
An unnamed structlog.get_logger() is named lite_bootstrap.instruments.logging_factory. structlog.stdlib.LoggerFactory guesses the name from the first frame outside structlog, and that frame is MemoryLoggerFactory.__call__.

Recommendation

Pursue it as an opt-in flag, False by default, with Sentry supported.

Who: services that log several lines per request or per message and do not need per-logger levels. Free, FastAPI and FastMCP. Raise ConfigurationError when the flag is set on Litestar or FastStream until per-logger levels have an answer there.
With Sentry: keep it supported. The saving is larger with Sentry (6.6 µs per info line) than without (4.7 µs), and the captured events match or improve on today's. LoggingInstrument builds the processor when sentry_dsn is set, read with getattr per ADR-0002, and passes sentry_logging_breadcrumb_level to it. SentryInstrument keeps LoggingIntegration for foreign loggers.
Cost to keep: a second processor chain, a copy of LoggingIntegration's event and breadcrumb shape that has to track sentry-sdk, and the docs changes above. The gain is small in absolute terms next to per-request costs elsewhere in the stack (OpenTelemetry alone is tens of µs per request), so this is worth it for log-heavy services and invisible for the rest.

Do the MemoryHandler removal separately and first, as a correctness change. It is not breaking for output, and it shrinks the bridge that the native path would sit next to. File the two factory bugs before either.

Benchmark and prototype

The native instrument subclasses LoggingInstrument and replaces _configure_structlog_loggers:

structlog.configure(
    processors=[
        structlog.processors.add_log_level,
        structlog.stdlib.add_logger_name,
        tracer_injection,
        config.logging_time_stamper or structlog.processors.TimeStamper(fmt="iso"),
        structlog.processors.StackInfoRenderer(),
        *([stash_exc_info] if sentry_capture else []),
        structlog.processors.format_exc_info,
        structlog.processors.UnicodeDecoder(),
        *config.logging_extra_processors,
        *([sentry_capture] if sentry_capture else []),
        structlog.processors.JSONRenderer(serializer=orjson.dumps),
    ],
    context_class=dict,
    logger_factory=structlog.BytesLoggerFactory(file),
    wrapper_class=structlog.make_filtering_bound_logger(config.logging_log_level),
    cache_logger_on_first_use=True,
)

structlog.stdlib.filter_by_level and PositionalArgumentsFormatter are dropped: the filtering bound logger does both. Foreign loggers keep _configure_foreign_loggers as is.

The "without MemoryHandler" variant is a MemoryLoggerFactory subclass whose __call__ attaches the StreamHandler with _RenderedLineFormatter directly.

Harness, one process per scenario:

class NullTransport(Transport):
    def capture_envelope(self, envelope):
        envelope.serialize()


def median_us(stmt, namespace, number, repeat):
    timer = timeit.Timer(stmt, globals=namespace)
    timer.timeit(number // 10)
    return statistics.median(timer.repeat(number=number, repeat=repeat)) / number * 1e6


def run(scenario):
    sys.stdout = open(os.devnull, "w")
    with_sentry = "+sentry" in scenario
    config = FreeConfig(
        sentry_dsn="https://public@o0.ingest.sentry.io/0" if with_sentry else None,
        sentry_additional_params={"transport": NullTransport},
    )
    # bootstrap LoggingInstrument, the no-MemoryHandler variant or the native one,
    # then SentryInstrument when with_sentry
    log = structlog.get_logger("bench.module")
    return {
        "info": median_us("log.info('did a thing', a=1)", ns, 20000, 15),
        "debug_filtered": median_us("log.debug('did a thing', a=1)", ns, 200000, 15),
        "error": median_us("log.error('plain error', a=2)", ns, 2000, 9),
        "exception": median_us("raise_and_log()", ns, 2000, 9),  # raise ValueError, log.exception('handled', b=3)
    }

Decision: not pursuing the native path. Closing.

The saving is about 4.7 µs per info line, or 6.6 µs with Sentry. That isn't worth keeping a second logging pipeline next to the stdlib one, along with a copy of sentry-sdk's event and breadcrumb format that would have to track sentry-sdk across majors (#380). Per-logger levels would also rule it out for Litestar and FastStream.

The intermediate step and the bugs found here are done: #358 fixed the two MemoryLoggerFactory bugs, and #377 replaced the MemoryHandler with a plain StreamHandler.

Navigation Menu

Repository navigation

Investigate a structlog-native output path that skips the stdlib bridge #342

Work

Findings: structlog-native output path, measured inside lite-bootstrap

Setup

Results (µs per call, median)

Sentry capture on the native path

`logging_record_filters` and per-logger levels

Dropping `MemoryHandler` (the intermediate step)

Two bugs found while measuring

Recommendation

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Investigate a structlog-native output path that skips the stdlib bridge #342

Description

Work

Activity

lesnik512 commented on Oct 28, 2026

Findings: structlog-native output path, measured inside lite-bootstrap

Setup

Results (µs per call, median)

Sentry capture on the native path

logging_record_filters and per-logger levels

Dropping MemoryHandler (the intermediate step)

Two bugs found while measuring

Recommendation

lesnik512 commented on Oct 29, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`logging_record_filters` and per-logger levels

Dropping `MemoryHandler` (the intermediate step)