Skip to content

[SVLS-9883] feat: add DD_LOG_LEVEL_BY_TARGET to set log levels per target #1413

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
lym953 wants to merge 6 commits into
base: main
Choose a base branch
from
Open

[SVLS-9883] feat: add DD_LOG_LEVEL_BY_TARGET to set log levels per target #1413

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
94d83cf
[SVLS-9883] Add DD_LOG_LEVEL_BY_TARGET to set log levels per target
lym953 Oct 16, 2026
9b19382
[SVLS-9883] fix: let DD_LOG_LEVEL_BY_TARGET turn on the HTTP library …
lym953 Oct 16, 2026
f08e336
[SVLS-9883] docs: document DD_LOG_LEVEL_BY_TARGET in the logger README
lym953 Oct 16, 2026
4442056
[SVLS-9883] docs: shorten the logger README
lym953 Oct 16, 2026
630212a
[SVLS-9883] docs: clarify the cold start example in the logger README
lym953 Oct 16, 2026
62a8bf7
[SVLS-9883] docs: reword the target name and invalid entry notes
lym953 Oct 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 14 additions & 4 deletions bottlecap/Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

9 changes: 5 additions & 4 deletions bottlecap/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,10 +95,11 @@ libdd-trace-normalization = "3.0.1"
libdd-trace-obfuscation = { version = "6.0.0", default-features = false }
libdd-trace-stats = { version = "7.0.0", default-features = false }
datadog-opentelemetry = { version = "0.5.2", default-features = false, features = ["_unstable_propagation"] }
dogstatsd = { git = "https://github.com/DataDog/serverless-components", rev = "8156757470d0843a99fc5d19809ae021cd2c7178", default-features = false }
datadog-fips = { git = "https://github.com/DataDog/serverless-components", rev = "8156757470d0843a99fc5d19809ae021cd2c7178", default-features = false }
datadog-agent-config = { git = "https://github.com/DataDog/serverless-components", rev = "8156757470d0843a99fc5d19809ae021cd2c7178", default-features = false }
datadog-agent-trace-sampler = { git = "https://github.com/DataDog/serverless-components", rev = "8156757470d0843a99fc5d19809ae021cd2c7178", default-features = false }
dogstatsd = { git = "https://github.com/DataDog/serverless-components", rev = "19270f5dcbc2480824bdec25a1d30899edb3e09e", default-features = false }
datadog-fips = { git = "https://github.com/DataDog/serverless-components", rev = "19270f5dcbc2480824bdec25a1d30899edb3e09e", default-features = false }
datadog-agent-config = { git = "https://github.com/DataDog/serverless-components", rev = "19270f5dcbc2480824bdec25a1d30899edb3e09e", default-features = false }
datadog-agent-trace-sampler = { git = "https://github.com/DataDog/serverless-components", rev = "19270f5dcbc2480824bdec25a1d30899edb3e09e", default-features = false }
datadog-serverless-logging = { git = "https://github.com/DataDog/serverless-components", rev = "19270f5dcbc2480824bdec25a1d30899edb3e09e", default-features = false }
libddwaf = { version = "1.28.1", git = "https://github.com/DataDog/libddwaf-rust", rev = "d1534a158d976bd4f747bf9fcc58e0712d2d17fc", default-features = false, features = ["serde"] }

[dev-dependencies]
Expand Down
1 change: 1 addition & 0 deletions bottlecap/LICENSE-3rdparty.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@ datadog-agent-trace-sampler,https://github.com/DataDog/serverless-components,Apa
datadog-fips,https://github.com/DataDog/serverless-components,Apache-2.0,The datadog-fips Authors
datadog-opentelemetry,https://github.com/DataDog/dd-trace-rs/tree/main/datadog-opentelemetry,Apache-2.0,Datadog Inc. <info@datadoghq.com>
datadog-protos,https://github.com/DataDog/saluki,Apache-2.0,The datadog-protos Authors
datadog-serverless-logging,https://github.com/DataDog/serverless-components,Apache-2.0,The datadog-serverless-logging Authors
ddsketch-agent,https://github.com/DataDog/saluki,Apache-2.0,The ddsketch-agent Authors
defmt,https://github.com/knurling-rs/defmt,MIT OR Apache-2.0,The Knurling-rs developers
defmt-macros,https://github.com/knurling-rs/defmt,MIT OR Apache-2.0,The Knurling-rs developers
Expand Down
19 changes: 12 additions & 7 deletions bottlecap/src/bin/bottlecap/main.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,7 @@ use bottlecap::{
},
};
use datadog_fips::reqwest_adapter::create_reqwest_client_builder;
use datadog_serverless_logging::{LEVEL_BY_TARGET_ENV_VAR, build_env_filter};
use decrypt::{
has_additional_endpoints_secrets, resolve_additional_endpoints_secrets, resolve_secrets,
};
Expand All @@ -107,7 +108,6 @@ use tokio::time::Instant;
use tokio::{sync::Mutex as TokioMutex, sync::mpsc::Sender};
use tokio_util::sync::CancellationToken;
use tracing::{debug, error, warn};
use tracing_subscriber::EnvFilter;
use ustr::Ustr;

fn resolve_dsm_service(
Expand Down Expand Up @@ -234,13 +234,15 @@ fn enable_logging_subsystem() {
)
.unwrap_or(LogLevel::Info);

let env_filter = format!(
"h2=off,hyper=off,reqwest=off,rustls=off,datadog-trace-mini-agent=off,{log_level:?}",
);
let (env_filter, invalid_entries) = build_env_filter(

@clifordshelton clifordshelton Oct 9, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The base filter turns off h2, hyper, reqwest and rustls for a reason, and with this change a DD_LOG_LEVEL_BY_TARGET entry can turn them back on. The extension reads its own stdout and sends it to Datadog as logs (see the "Be careful adding logs inside the logs pipeline" note in AGENTS.md). These HTTP crates log on every request, and that includes the requests the extension makes to send logs. So with hyper=debug, every flush creates more log lines, which get sent in the next flush, which creates more lines, and so on. That adds a lot of noise to the customer's CloudWatch and Datadog logs, and it costs them money.

Since this is an environment variable customers can set, I'd keep those four crates locked to off. You can skip any entry for them and add it to invalid_entries, so the user gets the same warning as for a typo. The cleanest place for this is probably the crate in serverless-components#180, for example by letting the caller pass a list of targets that can't be changed. That would mean making the change before #180 merges. The overrides_base_filter test that sets h2=debug would then change to check that the entry is rejected.

@lym953 lym953 Oct 16, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sometimes, we do want to turn on logging for them to debug network issues, for example, in SLES-3059. To avoid infinite loop, for logs from these modules, we can make them appear only in CloudWatch logs and don't send them to Datadog. I'm doing this in #1429

@lym953 lym953 Oct 16, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in #1429. Sometimes we do need these logs to debug network issues. That PR allows making them appear only in CloudWatch Logs but not Datadog to avoid infinite loops.

&format!(
"h2=off,hyper=off,reqwest=off,rustls=off,datadog-trace-mini-agent=off,{log_level:?}"
),
&std::env::var(LEVEL_BY_TARGET_ENV_VAR).unwrap_or_default(),
Comment on lines +237 to +241
)
.expect("could not parse log level in configuration");
let subscriber = tracing_subscriber::fmt::Subscriber::builder()
.with_env_filter(
EnvFilter::try_new(env_filter).expect("could not parse log level in configuration"),
)
.with_env_filter(env_filter)
.with_level(true)
.with_thread_names(false)
.with_thread_ids(false)
Expand All @@ -252,6 +254,9 @@ fn enable_logging_subsystem() {
.finish();
tracing::subscriber::set_global_default(subscriber).expect("setting default subscriber failed");

if !invalid_entries.is_empty() {
warn!("Ignoring invalid {LEVEL_BY_TARGET_ENV_VAR} entries: {invalid_entries:?}");
}
debug!("Logging subsystem enabled");
}

Expand Down
16 changes: 16 additions & 0 deletions bottlecap/src/logger/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# Log Levels by Target

`DD_LOG_LEVEL_BY_TARGET` turns on more logs for one part of the extension, without the debug logs of all other parts. It is mainly for Datadog engineers and support, to collect logs for a support ticket. The Datadog documentation site does not describe it.

Set these environment variables on the Lambda function:

| Environment variable | Value |
| --- | --- |
| `DD_LOG_LEVEL` | The level for all other logs, for example `info`. The default is `info`. |
| `DD_LOG_LEVEL_BY_TARGET` | A comma-separated list of `<target>=<level>` entries |

A level is `off`, `error`, `warn`, `info`, `debug` or `trace`. For example, if `DD_LOG_LEVEL` is `info` and `DD_LOG_LEVEL_BY_TARGET` is `cold_start_duration=debug`, the extension logs at `info` level as usual, and also logs the duration of each cold start step.

An entry matches each target that starts with its text. For an entry that is not valid, the extension logs a warning that names the entry, and skips it.

Use the targets that `mod.rs` defines, such as `cold_start_duration`. Other targets are module paths and crate names, which change when the code moves.
4 changes: 4 additions & 0 deletions bottlecap/src/logger/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,10 @@ use tracing_subscriber::registry::LookupSpan;

pub mod cold_start;

// `DD_LOG_LEVEL_BY_TARGET` matches a target by text prefix, so no target name may start with
// another one: an entry for `trace_flush` would also match `trace_flush_duration`.
// `README.md` in this folder documents `DD_LOG_LEVEL_BY_TARGET`.

/// Tracing target for the duration of each cold start step.
pub const COLD_START_DURATION: &str = "cold_start_duration";

Expand Down
Loading