What We Worked On
Codex keeps a local, persistent log sink: a SQLite database (logs_2.sqlite) written by a tracing layer in codex-rs/state/src/log_db.rs. It is a ring of fixed-size partitions — the layer inserts rows and prunes old ones to stay under a per-partition budget of 1,000 rows. That design is fine until something writes to it faster than a human ever reads it. Then the partition fills “in seconds” (the commit authors’ measurement) and the sink degrades into continuous insert-and-prune churn: every useful diagnostic is shoved out of the window by dependency noise before anyone can look at it.
That was issue #28224. This devblog walks the five commits that drained the firehose, in the order they landed. Three are upstream (jif@openai), pulled in through the Merge upstream/main into custom merge on 2026-06-25; two are local additions on the custom branch (philipbawn) — a regression test and the final INFO cap that closed the issue out.
Everything here touches one thing only: what gets persisted to the local SQLite ring. default_filter() is the log_db layer filter, so console output and RUST_LOG are unaffected, and the remote OpenTelemetry export plus all metrics are untouched.
The arc is worth naming up front, because each commit fixed a leak the previous one revealed:
- 191e6da9
Stop generating three log records per WebSocket event.
- 2e98d43a
Introduce a shared filter that turns the noisy targets
OFF. - 363f8f54
Discover the
OFFfilter didn’t actually catch bridged events, and reject them inside the sink. - 4fef76c4
Lock the behavior down with regression tests.
- 5a0e8bee
Cap the remaining default at INFO, because everything still defaulted to TRACE.
Commit 1 — Stop logging every Responses WebSocket event
91e6da9 (#29432) removed the single largest source. Every successful Responses WebSocket event was producing three local records: the full payload at TRACE, an OpenTelemetry log event, and an OpenTelemetry trace event (codex.websocket_event). On a busy thread that is the entire partition budget spent on routine success.
The fix removes the per-event TRACE payload line and stops emitting the codex.websocket_event log/trace pair, while keeping the counters, duration, and response-timing metrics, plus all parsing and error handling. The second hunk isn’t only a deletion — collapsing the per-arm error_message bookkeeping let the match arms fold together, which is why Binary/Close/Frame and Err(_) | None | Err(_) merge.
Commit 2 — Filter noisy targets from persistent logs
e98d43a (#29457). Before this, both the app-server and the TUI built the log_db layer filter inline and identically: Targets::new().with_default(Level::TRACE) — i.e. persist everything from everyone at TRACE. This commit factors that into a single default_filter() in log_db.rs and uses it as the place to encode exclusions: target=log (the tracing-log bridge for dependency log-crate records), codex_otel.log_only, and codex_otel.trace_safe (the two OTEL mirror targets that duplicate events already exported remotely) all go to OFF. Everything else still defaulted to TRACE at this point — that’s the loose end commit 5 ties off.
Commit 3 — Stop persisting bridged log events
63f8f54 (#29599) is the subtle one, and the reason the campaign needed more than a filter. with_target("log", OFF) looks like it should suppress the log-crate bridge — but it doesn’t.
tracing-log evaluates filters against the original dependency target (hyper, opentelemetry, …) before re-emitting the record with the tracing target log. By the time the record carries target = "log", the filter has already let it through under its real name. So high-volume dependency TRACE kept reaching SQLite despite the OFF directive.
The fix stops trusting the filter for this case and rejects the bridged record inside the sink’s on_event, before any formatting or queueing. (The opentelemetry_sdk DEBUG-timer guard just below it is pre-existing — the commit message notes those meta-events were once >30% of retained logs.)
Commit 4 — Add SQLite log filter regressions
fef76c4. With three behavioral fixes in place, the next local commit pinned them down. Two tests: one asserts default_filter() excludes the #28224 noisy targets (and still admits ordinary targets like codex_state); the other is the important end-to-end one — it wires the real sink behind a deliberately permissive with_default(TRACE) filter, emits a bridged target: "log" TRACE and a codex_state INFO, flushes, and asserts only the codex_state row survives in the DB. That test is what would catch a regression of commit 3 even if someone “fixed” the filter to allow log through again.
Commit 5 — Cap the persistent sink at INFO
a0e8bee is the loose end. After the three explicitly-OFF targets were handled, default_filter() still said .with_default(LevelFilter::TRACE). That meant the remaining write-amplification was ordinary TRACE/DEBUG chatter from everything that wasn’t named: dependencies like hyper_util and opentelemetry-*, and chatty internal targets — codex_api::sse, codex_mcp::connection_manager, codex_app_server::outgoing_message, codex_tui::markdown_stream. None of it is worth a partition slot in the persistent ring.
Capping the default at INFO keeps useful diagnostics (warnings, errors, and deliberate INFO breadcrumbs) while dropping the firehose. The regression test is updated from “ordinary targets keep TRACE” to the new contract: INFO is captured, TRACE is dropped.
The Net Result
default_filter() now reads, end to end:
…backstopped by the in-sink if metadata.target() == "log" { return; } guard for the bridge that the filter alone can’t catch. The persistent SQLite ring keeps INFO-and-above from real targets, with the bridged log firehose and the two OTEL mirror targets gone, and the busiest single source (per-event WebSocket logging) removed at the point of emission.
What Went Well
Each commit was a clean layer of the same onion. Reduce emission → filter targets → fix the filter’s blind spot → test it → cap the rest. No commit undid an earlier one; each addressed a leak the previous one exposed.
The bridge bug was diagnosed, not patched over. It would have been easy to shrug at “the OFF directive doesn’t work” and crank thresholds. Instead 63f8f54 names why (tracing-log filters on the original target) and fixes it at the right layer — inside the sink, where the bridged target is finally visible.
The regression test exercises the real sink, not just the filter object. sqlite_sink_drops_bridged_log_target_even_when_layer_filter_allows_it deliberately allows everything at the filter and still asserts the bridged row never lands in the DB. That is testing the guard, not the directive.
What Didn’t Go Well
The OFF directive was a false sense of security for a full commit. #29457 shipped believing target=log was disabled; #29599 had to follow up once it was clear bridged events still reached SQLite. The filter looked correct and wasn’t, which is the most expensive kind of wrong.
The default sat at TRACE through four commits. Three targets were named and silenced while the catch-all stayed at TRACE, so unnamed dependency and internal chatter kept amplifying writes until a0e8bee. The real fix — “stop persisting TRACE you never asked for” — was also the last to land.
The performance numbers are reported, not reproduced here. “Fills in seconds” and “>30% of retained logs” come from the commit authors’ measured environments; this writeup confirms the code changes from the diffs, not those magnitudes independently.
Takeaways
- 1A filter directive is a claim, not a guarantee — confirm where the record is actually evaluated.
with_target("log", OFF)reads as “nologevents,” but the bridge is filtered under its original target first. When a suppression rule sits upstream of a re-targeting layer, it can silently no-op. - 2The cheapest log to store is the one you never emit.
The biggest single win (
91e6da9) wasn’t a filter at all — it was deleting three records per WebSocket event at the source. Filter tuning is downstream cleanup; emission discipline is the dam. - 3A bounded, self-pruning sink turns volume into silent data loss.
A fixed 1,000-row partition doesn’t error when flooded — it quietly evicts the signal. “Why is my log empty right after the interesting thing happened?” is the failure mode, and it argues for filtering aggressively by default.
- 4Default to INFO for persistent local sinks; opt into TRACE deliberately.
with_default(TRACE)for a durable store means every dependency’s debug chatter is your problem forever. Make the floor INFO and let operators raise it. - 5Pin subtle filter behavior with an end-to-end test, not a unit assertion on the filter object.
The filter can pass
would_enablechecks and still be bypassed by a bridge; only a test that drives the real sink proves the row doesn’t land.
The Watercolor
I’d paint this one as a dam being built across a river at night, in four narrowing courses of stone, and I’d start with the water.
The first wash is a torrent — Payne’s grey and indigo poured wet-into-wet from the top of the sheet, no drawing underneath, just volume. That’s the log firehose: three records per heartbeat, a thousand-row pool that refills before the surface can settle. Down in the pool I’d drop a few flecks of cadmium and viridian — the useful diagnostics — and then let the grey roll right over them so you can barely find them again. That’s the whole problem in one gesture: the signal is in there, drowning.
Then the masonry, course by course, each a drier stroke than the last. The first stone is broad and confident — stop emitting — and the water visibly drops behind it. The second course looks solid but I’d paint one stone slightly the wrong color, a hair too warm, because it didn’t actually seal: the bridged water seeps under it, a thin ribbon of the original indigo finding the gap. The third course is the patient one — someone knelt down, found the seep, understood that the water was being measured by its old name before it was relabeled, and set the stone under the leak instead of over it. I’d render that stone in the most deliberate dry-brush on the page, every edge intentional.
The fourth course is almost nothing — a single low line of INFO-grey — but it’s what finally brings the water down to a readable level. Behind the finished dam the pool goes still and clear, and now you can see the cadmium and viridian flecks resting on the bottom, exactly where they were the whole time, finally legible.
The brushwork tells the honest story: bold and fast where deletion did the heavy lifting, one too-warm stone where the filter only looked sealed, and the calmest, most careful strokes saved for the small fix that mattered most. I’d title it The Pool Goes Clear When You Stop Pouring.