diff --git a/.cursor/rules/create-prd.mdc b/.cursor/rules/create-prd.mdc new file mode 100644 index 0000000000..5d99276b28 --- /dev/null +++ b/.cursor/rules/create-prd.mdc @@ -0,0 +1,60 @@ +--- +description: +globs: +alwaysApply: false +--- +# Rule: Generating a Product Requirements Document (PRD) + +## Goal + +To guide an AI assistant in creating a detailed Product Requirements Document (PRD) in Markdown format, based on an initial user prompt. The PRD should be clear, actionable, and suitable for a junior developer to understand and implement the feature. + +## Process + +1. **Receive Initial Prompt:** The user provides a brief description or request for a new feature or functionality. +2. **Ask Clarifying Questions:** Before writing the PRD, the AI *must* ask clarifying questions to gather sufficient detail. The goal is to understand the "what" and "why" of the feature, not necessarily the "how" (which the developer will figure out). +3. **Generate PRD:** Based on the initial prompt and the user's answers to the clarifying questions, generate a PRD using the structure outlined below. +4. **Save PRD:** Save the generated document as `prd-[feature-name].md` inside the `/tasks` directory. + +## Clarifying Questions (Examples) + +The AI should adapt its questions based on the prompt, but here are some common areas to explore: + +* **Problem/Goal:** "What problem does this feature solve for the user?" or "What is the main goal we want to achieve with this feature?" +* **Target User:** "Who is the primary user of this feature?" +* **Core Functionality:** "Can you describe the key actions a user should be able to perform with this feature?" +* **User Stories:** "Could you provide a few user stories? (e.g., As a [type of user], I want to [perform an action] so that [benefit].)" +* **Acceptance Criteria:** "How will we know when this feature is successfully implemented? What are the key success criteria?" +* **Scope/Boundaries:** "Are there any specific things this feature *should not* do (non-goals)?" +* **Data Requirements:** "What kind of data does this feature need to display or manipulate?" +* **Design/UI:** "Are there any existing design mockups or UI guidelines to follow?" or "Can you describe the desired look and feel?" +* **Edge Cases:** "Are there any potential edge cases or error conditions we should consider?" + +## PRD Structure + +The generated PRD should include the following sections: + +1. **Introduction/Overview:** Briefly describe the feature and the problem it solves. State the goal. +2. **Goals:** List the specific, measurable objectives for this feature. +3. **User Stories:** Detail the user narratives describing feature usage and benefits. +4. **Functional Requirements:** List the specific functionalities the feature must have. Use clear, concise language (e.g., "The system must allow users to upload a profile picture."). Number these requirements. +5. **Non-Goals (Out of Scope):** Clearly state what this feature will *not* include to manage scope. +6. **Design Considerations (Optional):** Link to mockups, describe UI/UX requirements, or mention relevant components/styles if applicable. +7. **Technical Considerations (Optional):** Mention any known technical constraints, dependencies, or suggestions (e.g., "Should integrate with the existing Auth module"). +8. **Success Metrics:** How will the success of this feature be measured? (e.g., "Increase user engagement by 10%", "Reduce support tickets related to X"). +9. **Open Questions:** List any remaining questions or areas needing further clarification. +## Target Audience + +Assume the primary reader of the PRD is a **junior developer**. Therefore, requirements should be explicit, unambiguous, and avoid jargon where possible. Provide enough detail for them to understand the feature's purpose and core logic. + +## Output + +* **Format:** Markdown (`.md`) +* **Location:** `/tasks/` +* **Filename:** `prd-[feature-name].md` + +## Final instructions + +1. Do NOT start implementing the PRD +2. Make sure to ask the user clarifying questions +3. Take the user's answers to the clarifying questions and improve the PRD diff --git a/.cursor/rules/generate-tasks.mdc b/.cursor/rules/generate-tasks.mdc new file mode 100644 index 0000000000..d6bf3e3d89 --- /dev/null +++ b/.cursor/rules/generate-tasks.mdc @@ -0,0 +1,63 @@ +--- +description: +globs: +alwaysApply: false +--- +# Rule: Generating a Task List from a PRD + +## Goal + +To guide an AI assistant in creating a detailed, step-by-step task list in Markdown format based on an existing Product Requirements Document (PRD). The task list should guide a developer through implementation. + +## Output + +- **Format:** Markdown (`.md`) +- **Location:** `/tasks/` +- **Filename:** `tasks-[prd-file-name].md` (e.g., `tasks-prd-user-profile-editing.md`) +## Process + +1. **Receive PRD Reference:** The user points the AI to a specific PRD file +2. **Analyze PRD:** The AI reads and analyzes the functional requirements, user stories, and other sections of the specified PRD. +3. **Phase 1: Generate Parent Tasks:** Based on the PRD analysis, create the file and generate the main, high-level tasks required to implement the feature. Use your judgement on how many high-level tasks to use. It's likely to be about 5. Present these tasks to the user in the specified format (without sub-tasks yet). Inform the user: "I have generated the high-level tasks based on the PRD. Ready to generate the sub-tasks? Respond with 'Go' to proceed." +4. **Wait for Confirmation:** Pause and wait for the user to respond with "Go". +5. **Phase 2: Generate Sub-Tasks:** Once the user confirms, break down each parent task into smaller, actionable sub-tasks necessary to complete the parent task. Ensure sub-tasks logically follow from the parent task and cover the implementation details implied by the PRD. +6. **Identify Relevant Files:** Based on the tasks and PRD, identify potential files that will need to be created or modified. List these under the `Relevant Files` section, including corresponding test files if applicable. +7. **Generate Final Output:** Combine the parent tasks, sub-tasks, relevant files, and notes into the final Markdown structure. +8. **Save Task List:** Save the generated document in the `/tasks/` directory with the filename `tasks-[prd-file-name].md`, where `[prd-file-name]` matches the base name of the input PRD file (e.g., if the input was `prd-user-profile-editing.md`, the output is `tasks-prd-user-profile-editing.md`). + +## Output Format + +The generated task list _must_ follow this structure: + +```markdown +## Relevant Files + +- `path/to/potential/file1.ts` - Brief description of why this file is relevant (e.g., Contains the main component for this feature). +- `path/to/file1.test.ts` - Unit tests for `file1.ts`. +- `path/to/another/file.tsx` - Brief description (e.g., API route handler for data submission). +- `path/to/another/file.test.tsx` - Unit tests for `another/file.tsx`. +- `lib/utils/helpers.ts` - Brief description (e.g., Utility functions needed for calculations). +- `lib/utils/helpers.test.ts` - Unit tests for `helpers.ts`. + +### Notes + +- Unit tests should typically be placed alongside the code files they are testing (e.g., `MyComponent.tsx` and `MyComponent.test.tsx` in the same directory). +- Use `npx jest [optional/path/to/test/file]` to run tests. Running without a path executes all tests found by the Jest configuration. + +## Tasks + +- [ ] 1.0 Parent Task Title + - [ ] 1.1 [Sub-task description 1.1] + - [ ] 1.2 [Sub-task description 1.2] +- [ ] 2.0 Parent Task Title + - [ ] 2.1 [Sub-task description 2.1] +- [ ] 3.0 Parent Task Title (may not require sub-tasks if purely structural or configuration) +``` + +## Interaction Model + +The process explicitly requires a pause after generating parent tasks to get user confirmation ("Go") before proceeding to generate the detailed sub-tasks. This ensures the high-level plan aligns with user expectations before diving into details. + +## Target Audience + +Assume the primary reader of the task list is a **junior developer** who will implement the feature. diff --git a/.cursor/rules/process-task-list.mdc b/.cursor/rules/process-task-list.mdc new file mode 100644 index 0000000000..7312cd52ec --- /dev/null +++ b/.cursor/rules/process-task-list.mdc @@ -0,0 +1,37 @@ +--- +description: +globs: +alwaysApply: false +--- +# Task List Management + +Guidelines for managing task lists in markdown files to track progress on completing a PRD +## Task Implementation +- **One sub-task at a time:** Do **NOT** start the next sub‑task until the previous one is finished. +- **Completion protocol:** + 1. When you finish a **sub‑task**, immediately mark it as completed by changing `[ ]` to `[x]`. + 2. If **all** subtasks underneath a parent task are now `[x]`, also mark the **parent task** as completed. +- Stop after each parent task and wait for the users go ahead. + +## Task List Maintenance + +1. **Update the task list as you work:** + - Mark tasks and subtasks as completed (`[x]`) per the protocol above. + - Add new tasks as they emerge. + +2. **Maintain the “Relevant Files” section:** + - List every file created or modified. + - Give each file a one‑line description of its purpose. + +## AI Instructions + +When working with task lists, the AI must: + +1. Regularly update the task list file after finishing any significant work. +2. Follow the completion protocol: + - Mark each finished **sub‑task** `[x]`. + - Mark the **parent task** `[x]` once **all** its subtasks are `[x]`. +3. Add newly discovered tasks. +4. Keep “Relevant Files” accurate and up to date. +5. Before starting work, check which sub‑task is next. +6. After implementing a sub‑task, update the file and then pause for user approval. diff --git a/Cargo.lock b/Cargo.lock index 010d0a3463..139d8e5aad 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -391,6 +391,12 @@ version = "1.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "baf1de4339761588bc0619e3cbc0120ee582ebb74b53b4efbf79117bd2da40fd" +[[package]] +name = "cfg_aliases" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "613afe47fcd5fac7ccf1db93babcb082c5994d996f20b8b159f2ad1658eb5724" + [[package]] name = "chrono" version = "0.4.38" @@ -570,6 +576,15 @@ dependencies = [ "cfg-if", ] +[[package]] +name = "crossbeam-channel" +version = "0.5.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "82b8f8f868b36967f9606790d1903570de9ceaf870a7bf9fbbd3016d636a2cb2" +dependencies = [ + "crossbeam-utils", +] + [[package]] name = "crossbeam-deque" version = "0.8.5" @@ -626,6 +641,16 @@ dependencies = [ "memchr", ] +[[package]] +name = "ctrlc" +version = "3.4.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "46f93780a459b7d656ef7f071fe699c4d3d2cb201c4b24d085b6ddc505276e73" +dependencies = [ + "nix", + "windows-sys 0.59.0", +] + [[package]] name = "curl" version = "0.4.46" @@ -962,6 +987,15 @@ dependencies = [ "percent-encoding", ] +[[package]] +name = "fsevent-sys" +version = "4.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "76ee7a02da4d231650c7cea31349b889be2f45ddb3ef3032d2ec8185f6313fd2" +dependencies = [ + "libc", +] + [[package]] name = "funty" version = "2.0.0" @@ -1558,6 +1592,26 @@ dependencies = [ "regex", ] +[[package]] +name = "inotify" +version = "0.9.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8069d3ec154eb856955c1c0fbffefbf5f3c40a104ec912d4797314c1801abff" +dependencies = [ + "bitflags 1.3.2", + "inotify-sys", + "libc", +] + +[[package]] +name = "inotify-sys" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e05c02b5e89bff3b946cedeca278abc628fe811e604f027c45a8aa3cf793d0eb" +dependencies = [ + "libc", +] + [[package]] name = "inout" version = "0.1.3" @@ -1668,6 +1722,26 @@ dependencies = [ "wasm-bindgen", ] +[[package]] +name = "kqueue" +version = "1.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "eac30106d7dce88daf4a3fcb4879ea939476d5074a9b7ddd0fb97fa4bed5596a" +dependencies = [ + "kqueue-sys", + "libc", +] + +[[package]] +name = "kqueue-sys" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ed9625ffda8729b85e45cf04090035ac368927b8cebc34898e7c120f52e4838b" +dependencies = [ + "bitflags 1.3.2", + "libc", +] + [[package]] name = "lazy_static" version = "1.5.0" @@ -1839,6 +1913,18 @@ dependencies = [ "adler", ] +[[package]] +name = "mio" +version = "0.8.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a4a650543ca06a924e8b371db273b2756685faae30f8487da1b56505a8f78b0c" +dependencies = [ + "libc", + "log", + "wasi 0.11.0+wasi-snapshot-preview1", + "windows-sys 0.48.0", +] + [[package]] name = "mio" version = "1.0.2" @@ -1881,6 +1967,18 @@ version = "1.0.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "650eef8c711430f1a879fdd01d4745a7deea475becfb90269c06775983bbf086" +[[package]] +name = "nix" +version = "0.30.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "74523f3a35e05aba87a1d978330aef40f67b0304ac79c1c00b294c9830543db6" +dependencies = [ + "bitflags 2.6.0", + "cfg-if", + "cfg_aliases", + "libc", +] + [[package]] name = "nom" version = "7.1.3" @@ -1910,6 +2008,25 @@ version = "0.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "61807f77802ff30975e01f4f071c8ba10c022052f98b3294119f3e615d13e5be" +[[package]] +name = "notify" +version = "6.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6205bd8bb1e454ad2e27422015fb5e4f2bcc7e08fa8f27058670d208324a4d2d" +dependencies = [ + "bitflags 2.6.0", + "crossbeam-channel", + "filetime", + "fsevent-sys", + "inotify", + "kqueue", + "libc", + "log", + "mio 0.8.11", + "walkdir", + "windows-sys 0.48.0", +] + [[package]] name = "num-conv" version = "0.1.0" @@ -2706,6 +2823,7 @@ dependencies = [ "clap", "clap_complete", "console", + "ctrlc", "curl", "data-encoding", "dirs", @@ -2726,6 +2844,7 @@ dependencies = [ "mac-process-info", "magic_string", "mockito", + "notify", "open", "openssl-probe", "parking_lot", @@ -2734,6 +2853,7 @@ dependencies = [ "prettytable-rs", "proguard", "r2d2", + "rand", "rayon", "regex", "rstest", @@ -3276,7 +3396,7 @@ dependencies = [ "backtrace", "bytes", "libc", - "mio", + "mio 1.0.2", "parking_lot", "pin-project-lite", "socket2", diff --git a/Cargo.toml b/Cargo.toml index 5aa663d144..652dae44c4 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -28,6 +28,7 @@ clap = { version = "4.1.6", default-features = false, features = [ ] } clap_complete = "4.4.3" console = "0.15.5" +ctrlc = "3.4.0" curl = { version = "0.4.46", features = ["static-curl", "static-ssl"] } dirs = "4.0.0" dotenvy = "0.15.7" @@ -46,6 +47,7 @@ java-properties = "1.4.1" lazy_static = "1.4.0" libc = "0.2.139" log = { version = "0.4.17", features = ["std"] } +notify = "6.1.1" open = "3.2.0" parking_lot = "0.12.1" percent-encoding = "2.2.0" @@ -53,6 +55,7 @@ plist = "1.4.0" prettytable-rs = "0.10.0" proguard = { version = "5.0.0", features = ["uuid"] } r2d2 = "0.8.10" +rand = "0.8.5" rayon = "1.6.1" regex = "1.7.3" runas = "1.0.0" diff --git a/src/api/envelopes_api.rs b/src/api/envelopes_api.rs index 80d2ad94b5..09b164c84d 100644 --- a/src/api/envelopes_api.rs +++ b/src/api/envelopes_api.rs @@ -37,4 +37,20 @@ impl EnvelopesApi { .send()? .into_result() } + + /// Send a raw envelope (for logs protocol compliance) + pub fn send_raw_envelope(&self, envelope_bytes: Vec) -> ApiResult { + let url = self.dsn.envelope_api_url(); + let auth = self.dsn.to_auth(Some(USER_AGENT)); + debug!( + "Sending raw envelope:\n{}", + String::from_utf8_lossy(&envelope_bytes) + ); + self.api + .request(Method::Post, url.as_str(), None)? + .with_header("X-Sentry-Auth", &auth.to_string())? + .with_body(envelope_bytes) + .send()? + .into_result() + } } diff --git a/src/commands/logs/mod.rs b/src/commands/logs/mod.rs index ce0c2cdc4e..9ebe4f994f 100644 --- a/src/commands/logs/mod.rs +++ b/src/commands/logs/mod.rs @@ -1,6 +1,8 @@ mod list; +mod tail; use self::list::ListLogsArgs; +use self::tail::TailLogsArgs; use super::derive_parser::{SentryCLI, SentryCLICommand}; use anyhow::Result; use clap::ArgMatches; @@ -10,6 +12,7 @@ const BETA_WARNING: &str = "[BETA] The \"logs\" command is in beta. The command to breaking changes, including removal, in any Sentry CLI release."; const LIST_ABOUT: &str = "List logs from your organization"; +const TAIL_ABOUT: &str = "Monitor log files in real-time and send entries to Sentry"; #[derive(Args)] pub(super) struct LogsArgs { @@ -30,8 +33,16 @@ enum LogsSubcommand { Query and filter log entries from your Sentry projects. \ Supports filtering by log level and custom queries.\n\n\ {BETA_WARNING}") -)] + )] List(ListLogsArgs), + #[command(about = format!("[BETA] {TAIL_ABOUT}"))] + #[command(long_about = format!("{TAIL_ABOUT}. \ + Continuously monitor a log file for new entries and send them to Sentry \ + as structured logging events. Supports common log formats including nginx \ + and Apache Common Log Format.\n\n\ + {BETA_WARNING}") + )] + Tail(TailLogsArgs), } pub(super) fn make_command(command: Command) -> Command { @@ -47,5 +58,6 @@ pub(super) fn execute(_: &ArgMatches) -> Result<()> { match subcommand { LogsSubcommand::List(args) => list::execute(args), + LogsSubcommand::Tail(args) => tail::execute(args), } } diff --git a/src/commands/logs/tail.rs b/src/commands/logs/tail.rs new file mode 100644 index 0000000000..8a3c3222e3 --- /dev/null +++ b/src/commands/logs/tail.rs @@ -0,0 +1,430 @@ +use anyhow::Result; +use clap::Args; +use log::{debug, info, warn}; +use std::path::PathBuf; +use std::time::Duration; + +use crate::utils::batching::{AdaptiveBatchingConfig, LogBatch}; +use crate::utils::file_watcher::{ + setup_signal_handlers, FileEvent, LogFileWatcher, PositionTracker, +}; +use crate::utils::log_parsing::{ + create_parser, detect_log_format, LogEntry, LogFormat as ParserLogFormat, +}; +use crate::utils::log_transmission::{LogTransmitter, TransmissionRateLimiter}; +use crate::utils::memory_monitor::{estimate_entry_size, MemoryMonitor}; +use crate::utils::sampling::PrioritySampler; + +/// Arguments for the tail logs command +#[derive(Args)] +pub(super) struct TailLogsArgs { + #[arg(help = "Path to the log file to monitor")] + file: PathBuf, + + #[arg(short = 'o', long = "org")] + #[arg(help = "The organization ID or slug.")] + org: Option, + + #[arg(short = 'p', long = "project")] + #[arg(help = "The project ID (slug not supported).")] + project: Option, + + #[arg(long = "batch-size", default_value = "100")] + #[arg(help = "Number of log entries to batch before sending to Sentry (1-1000)")] + #[arg(value_parser = clap::value_parser!(u32).range(1..=1000))] + batch_size: u32, + + #[arg(long = "batch-timeout", default_value = "5")] + #[arg(help = "Timeout in seconds for sending partial batches")] + #[arg(value_parser = clap::value_parser!(u64).range(1..=300))] + batch_timeout: u64, + + #[arg(long = "rate-limit", default_value = "1000")] + #[arg(help = "Maximum number of log entries to send per minute")] + #[arg(value_parser = clap::value_parser!(u32).range(1..=10000))] + rate_limit: u32, + + #[arg(long = "memory-limit", default_value = "50")] + #[arg(help = "Maximum memory usage in MB for buffering log entries")] + memory_limit: usize, + + #[arg(long = "sampling-rate", default_value = "1.0")] + #[arg(help = "Sampling rate for high-volume logs (0.0-1.0, 1.0 = no sampling)")] + sampling_rate: f64, + + #[arg(long = "format")] + #[arg(help = "Log format to parse (auto-detect if not specified)")] + #[arg(value_enum)] + format: Option, +} + +/// Supported log formats for parsing +#[derive(clap::ValueEnum, Clone, Debug, Default)] +pub enum LogFormat { + /// Auto-detect format based on file content + #[default] + Auto, + /// nginx default log format + Nginx, + /// Apache Common Log Format + Apache, + /// Plain text logs (fallback) + Plain, +} + +pub(super) fn execute(args: TailLogsArgs) -> Result<()> { + use crate::config::Config; + + // Validate that the file exists and is readable + if !args.file.exists() { + anyhow::bail!("Log file does not exist: {}", args.file.display()); + } + + if !args.file.is_file() { + anyhow::bail!("Path is not a file: {}", args.file.display()); + } + + // Validate organization and project configuration + let config = Config::current(); + let (default_org, default_project) = config.get_org_and_project_defaults(); + + let org = args.org.as_ref().or(default_org.as_ref()).ok_or_else(|| { + anyhow::anyhow!( + "No organization specified. Please specify an organization using the --org argument." + ) + })?; + + let project = args + .project + .as_ref() + .or(default_project.as_ref()) + .ok_or_else(|| { + anyhow::anyhow!("No project specified. Use --project or set a default in config.") + })?; + + // Read file permissions to ensure we can read it + std::fs::File::open(&args.file) + .map_err(|e| anyhow::anyhow!("Cannot read log file {}: {}", args.file.display(), e))?; + + info!("Starting to tail log file: {}", args.file.display()); + info!("Organization: {org}"); + info!("Project: {project}"); + info!("Batch size: {}", args.batch_size); + info!("Batch timeout: {}s", args.batch_timeout); + info!( + "Format: {:?}", + args.format.as_ref().unwrap_or(&LogFormat::Auto) + ); + + // Set up signal handling for graceful shutdown + let shutdown_receiver = setup_signal_handlers()?; + + // Initialize file watcher and position tracker + let file_watcher = LogFileWatcher::new(&args.file)?; + let mut position_tracker = PositionTracker::new(&args.file)?; + + // Initialize adaptive batching with memory awareness + let adaptive_config = AdaptiveBatchingConfig { + min_batch_size: 10, + max_batch_size: (args.batch_size as usize * 2).min(1000), + + min_timeout: Duration::from_secs(1), + max_timeout: Duration::from_secs(args.batch_timeout * 2), + recent_flush_times: Vec::new(), + max_history: 10, + }; + + let mut log_batch = LogBatch::new_adaptive( + args.batch_size as usize, + Duration::from_secs(args.batch_timeout), + adaptive_config, + ); + + // Determine log format + let log_format = match &args.format { + Some(LogFormat::Auto) | None => { + // Auto-detect format by reading first few lines + info!("Auto-detecting log format..."); + let sample_lines = read_sample_lines(&args.file, 10)?; + let detected_format = detect_log_format(&sample_lines); + info!("Detected log format: {}", detected_format.name()); + detected_format + } + Some(LogFormat::Nginx) => ParserLogFormat::Nginx, + Some(LogFormat::Apache) => ParserLogFormat::Apache, + Some(LogFormat::Plain) => ParserLogFormat::Plain, + }; + + // Create parser for the determined format + let parser = create_parser(log_format); + + // Initialize Sentry log transmitter + let log_transmitter = LogTransmitter::new(org.clone(), project.clone())?; + let mut rate_limiter = TransmissionRateLimiter::new(args.rate_limit); + + // Initialize performance monitoring and optimization features + let memory_monitor = MemoryMonitor::new(args.memory_limit); + let mut sampler = if args.sampling_rate < 1.0 { + Some(PrioritySampler::new(args.rate_limit as f64 / 60.0)) // Convert per-minute to per-second + } else { + None + }; + + let poll_interval = Duration::from_millis(1000); + + info!("File monitoring started. Press Ctrl+C to stop."); + + loop { + // Check for shutdown signal + if shutdown_receiver.try_recv().is_ok() { + info!("Shutdown signal received, flushing remaining logs..."); + + // Flush any remaining log entries + if !log_batch.is_empty() { + let remaining_entries = log_batch.flush(); + info!("Flushing {} remaining log entries", remaining_entries.len()); + if let Err(e) = + send_log_entries(&log_transmitter, &mut rate_limiter, remaining_entries) + { + warn!("Failed to send final batch: {}", e); + } + } + + break; + } + + // Check for file system events + match file_watcher.check_events(poll_interval)? { + Some(FileEvent::DataWritten) => { + // File has new data, read new lines + let new_lines = position_tracker.read_new_lines()?; + + for line in new_lines { + debug!("New log line: {}", line); + + // Check memory usage before processing + let entry_size = estimate_entry_size(&line); + if !memory_monitor.record_log_entry(entry_size) { + warn!("Memory limit exceeded, dropping log entry"); + continue; + } + + // Parse log format and create structured log entry + match parser.parse_line(&line) { + Ok(log_entry) => { + debug!("Parsed log entry: {:?}", log_entry); + + // Apply sampling if configured + let should_process = if let Some(ref mut sampler) = sampler { + let level_str = + log_entry.level.as_ref().map(|l| l.to_sentry_level()); + sampler.should_sample(level_str) + } else { + true + }; + + if should_process { + let should_flush = + log_batch.add_entry(serialize_log_entry(&log_entry)?); + + if should_flush { + let entries = log_batch.flush(); + info!( + "Sending batch of {} log entries to Sentry", + entries.len() + ); + if let Err(e) = send_log_entries_with_monitoring( + &log_transmitter, + &mut rate_limiter, + entries, + &memory_monitor, + ) { + warn!("Failed to send log batch: {}", e); + } + } + } else { + debug!("Entry dropped by sampling"); + } + } + Err(e) => { + warn!("Failed to parse log line '{}': {}", line, e); + // Fall back to plain text + let should_process = if let Some(ref mut sampler) = sampler { + sampler.should_sample(Some("info")) // Default to info level for unparsed + } else { + true + }; + + if should_process { + let should_flush = log_batch.add_entry(line); + + if should_flush { + let entries = log_batch.flush(); + info!( + "Sending batch of {} log entries to Sentry", + entries.len() + ); + if let Err(e) = send_log_entries_with_monitoring( + &log_transmitter, + &mut rate_limiter, + entries, + &memory_monitor, + ) { + warn!("Failed to send log batch: {}", e); + } + } + } + } + } + } + } + + Some(FileEvent::Deleted) => { + warn!("Log file was deleted, stopping monitoring"); + break; + } + Some(FileEvent::Moved) => { + warn!("Log file was moved/renamed, attempting to continue monitoring"); + // For log rotation, we might want to restart watching + } + Some(FileEvent::Created) => { + info!("Log file was created/recreated"); + position_tracker = PositionTracker::new(&args.file)?; + } + None => { + // No file events, check if batch should be flushed due to timeout + if log_batch.should_flush() && !log_batch.is_empty() { + let entries = log_batch.flush(); + info!( + "Timeout flush: sending batch of {} log entries to Sentry", + entries.len() + ); + if let Err(e) = send_log_entries(&log_transmitter, &mut rate_limiter, entries) { + warn!("Failed to send timeout batch: {}", e); + } + } + } + } + + // Periodic check for new data even without file system events + let new_bytes = position_tracker.check_for_new_data()?; + if new_bytes > 0 { + let new_lines = position_tracker.read_new_lines()?; + + for line in new_lines { + debug!("New log line (polling): {}", line); + + // Parse log format and create structured log entry + match parser.parse_line(&line) { + Ok(log_entry) => { + debug!("Parsed log entry: {:?}", log_entry); + let should_flush = log_batch.add_entry(serialize_log_entry(&log_entry)?); + + if should_flush { + let entries = log_batch.flush(); + info!("Sending batch of {} log entries to Sentry", entries.len()); + if let Err(e) = + send_log_entries(&log_transmitter, &mut rate_limiter, entries) + { + warn!("Failed to send log batch: {}", e); + } + } + } + Err(e) => { + warn!("Failed to parse log line '{}': {}", line, e); + // Fall back to plain text + let should_flush = log_batch.add_entry(line); + + if should_flush { + let entries = log_batch.flush(); + info!("Sending batch of {} log entries to Sentry", entries.len()); + if let Err(e) = + send_log_entries(&log_transmitter, &mut rate_limiter, entries) + { + warn!("Failed to send log batch: {}", e); + } + } + } + } + } + } + } + + info!("Log file monitoring stopped"); + Ok(()) +} + +/// Read sample lines from a file for format auto-detection +fn read_sample_lines(file_path: &PathBuf, max_lines: usize) -> Result> { + use std::fs::File; + use std::io::{BufRead as _, BufReader}; + + let file = File::open(file_path)?; + let reader = BufReader::new(file); + + let mut lines = Vec::new(); + for line in reader.lines().take(max_lines) { + let line = line?; + if !line.trim().is_empty() { + lines.push(line); + } + } + + Ok(lines) +} + +/// Serialize a LogEntry to JSON string for batching +fn serialize_log_entry(entry: &LogEntry) -> Result { + serde_json::to_string(entry) + .map_err(|e| anyhow::anyhow!("Failed to serialize log entry: {}", e)) +} + +/// Send log entries to Sentry with rate limiting +fn send_log_entries( + transmitter: &LogTransmitter, + rate_limiter: &mut TransmissionRateLimiter, + entries: Vec, +) -> Result<()> { + if !rate_limiter.can_send() { + let (current, max) = rate_limiter.get_status(); + warn!( + "Rate limit exceeded: {}/{} events this minute. Dropping {} log entries.", + current, + max, + entries.len() + ); + return Ok(()); + } + + match transmitter.send_log_batch(entries) { + Ok(event_ids) => { + debug!( + "Successfully sent {} log entries to Sentry", + event_ids.len() + ); + Ok(()) + } + Err(e) => { + warn!("Failed to transmit logs to Sentry: {}", e); + Err(e) + } + } +} + +/// Send log entries to Sentry with rate limiting and memory monitoring +fn send_log_entries_with_monitoring( + transmitter: &LogTransmitter, + rate_limiter: &mut TransmissionRateLimiter, + entries: Vec, + memory_monitor: &MemoryMonitor, +) -> Result<()> { + // Calculate memory that will be freed + let freed_bytes: usize = entries.iter().map(|e| estimate_entry_size(e)).sum(); + + let result = send_log_entries(transmitter, rate_limiter, entries); + + // Update memory tracking after transmission + memory_monitor.record_flush(freed_bytes); + + result +} diff --git a/src/utils/batching.rs b/src/utils/batching.rs new file mode 100644 index 0000000000..1da3f6827d --- /dev/null +++ b/src/utils/batching.rs @@ -0,0 +1,214 @@ +#![allow(clippy::allow_attributes)] + +use log::debug; +use std::time::{Duration, Instant}; + +/// A batching utility for collecting log entries before sending to Sentry +#[derive(Debug)] +pub struct LogBatch { + entries: Vec, + max_size: usize, + timeout: Duration, + last_flush: Instant, + adaptive_config: AdaptiveBatchingConfig, +} + +/// Configuration for adaptive batching behavior +#[derive(Debug, Clone)] + +pub struct AdaptiveBatchingConfig { + /// Minimum batch size (always flush at least this many) + pub min_batch_size: usize, + /// Maximum batch size (never exceed this) + pub max_batch_size: usize, + + /// Minimum timeout (for high-volume periods) + pub min_timeout: Duration, + /// Maximum timeout (for low-volume periods) + pub max_timeout: Duration, + /// Track recent flush rates for adaptation + pub recent_flush_times: Vec, + /// Maximum number of recent flush times to track + pub max_history: usize, +} + +impl Default for AdaptiveBatchingConfig { + fn default() -> Self { + AdaptiveBatchingConfig { + min_batch_size: 10, + max_batch_size: 1000, + + min_timeout: Duration::from_secs(1), + max_timeout: Duration::from_secs(30), + recent_flush_times: Vec::new(), + max_history: 10, + } + } +} + +impl LogBatch { + /// Create a new log batch with specified maximum size and timeout + #[allow(dead_code)] + pub fn new(max_size: usize, timeout: Duration) -> Self { + LogBatch { + entries: Vec::with_capacity(max_size), + max_size, + timeout, + last_flush: Instant::now(), + adaptive_config: AdaptiveBatchingConfig::default(), + } + } + + /// Create a new adaptive log batch with custom configuration + pub fn new_adaptive( + max_size: usize, + timeout: Duration, + adaptive_config: AdaptiveBatchingConfig, + ) -> Self { + LogBatch { + entries: Vec::with_capacity(max_size.min(adaptive_config.max_batch_size)), + max_size, + timeout, + last_flush: Instant::now(), + adaptive_config, + } + } + + /// Add a log entry to the batch + /// Returns true if the batch should be flushed (due to size or timeout) + pub fn add_entry(&mut self, entry: String) -> bool { + self.entries.push(entry); + + debug!( + "Added log entry to batch ({}/{})", + self.entries.len(), + self.max_size + ); + + self.should_flush() + } + + /// Check if the batch should be flushed due to size or timeout + pub fn should_flush(&self) -> bool { + let current_timeout = self.get_adaptive_timeout(); + let size_threshold = self.get_adaptive_batch_size(); + + self.entries.len() >= size_threshold || self.last_flush.elapsed() >= current_timeout + } + + /// Get all entries and clear the batch + pub fn flush(&mut self) -> Vec { + let entries = std::mem::take(&mut self.entries); + let now = Instant::now(); + + // Record flush time for adaptive behavior + self.adaptive_config.recent_flush_times.push(now); + if self.adaptive_config.recent_flush_times.len() > self.adaptive_config.max_history { + self.adaptive_config.recent_flush_times.remove(0); + } + + self.last_flush = now; + + debug!("Flushed batch with {} entries", entries.len()); + + entries + } + + /// Calculate adaptive timeout based on recent flush patterns + fn get_adaptive_timeout(&self) -> Duration { + if self.adaptive_config.recent_flush_times.len() < 2 { + return self.timeout; + } + + // Calculate average time between flushes + let recent_times = &self.adaptive_config.recent_flush_times; + let mut total_interval = Duration::from_secs(0); + for i in 1..recent_times.len() { + total_interval += recent_times[i].duration_since(recent_times[i - 1]); + } + + let avg_interval = total_interval / (recent_times.len() - 1) as u32; + + // Adapt timeout based on flush frequency + // If flushing frequently (high volume), use shorter timeout + // If flushing infrequently (low volume), use longer timeout + if avg_interval < Duration::from_secs(2) { + // High volume - decrease timeout + self.adaptive_config.min_timeout + } else if avg_interval > Duration::from_secs(10) { + // Low volume - increase timeout + self.adaptive_config.max_timeout + } else { + // Normal volume - use base timeout + self.timeout + } + } + + /// Calculate adaptive batch size based on recent patterns + fn get_adaptive_batch_size(&self) -> usize { + if self.adaptive_config.recent_flush_times.len() < 2 { + return self.max_size; + } + + // If we're flushing very frequently, increase batch size to be more efficient + let recent_times = &self.adaptive_config.recent_flush_times; + if recent_times.len() >= 3 { + let last_intervals: Vec<_> = recent_times + .windows(2) + .take(3) + .map(|w| w[1].duration_since(w[0])) + .collect(); + + let avg_recent_interval = + last_intervals.iter().sum::() / last_intervals.len() as u32; + + if avg_recent_interval < Duration::from_secs(1) { + // Very high volume - increase batch size for efficiency + (self.max_size * 2).min(self.adaptive_config.max_batch_size) + } else if avg_recent_interval > Duration::from_secs(20) { + // Very low volume - decrease batch size for responsiveness + self.adaptive_config.min_batch_size + } else { + self.max_size + } + } else { + self.max_size + } + } + + /// Check if the batch is empty + pub fn is_empty(&self) -> bool { + self.entries.is_empty() + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_batch_size_limit() { + let mut batch = LogBatch::new(2, Duration::from_secs(60)); + + assert!(!batch.add_entry("entry 1".into())); + assert!(batch.add_entry("entry 2".into())); + + let entries = batch.flush(); + assert_eq!(entries.len(), 2); + assert_eq!(entries[0], "entry 1"); + assert_eq!(entries[1], "entry 2"); + assert!(batch.is_empty()); + } + + #[test] + fn test_batch_timeout() { + let mut batch = LogBatch::new(10, Duration::from_millis(1)); + + batch.add_entry("entry 1".into()); + + // Wait for timeout + std::thread::sleep(Duration::from_millis(2)); + + assert!(batch.should_flush()); + } +} diff --git a/src/utils/file_watcher/mod.rs b/src/utils/file_watcher/mod.rs new file mode 100644 index 0000000000..e2061772e1 --- /dev/null +++ b/src/utils/file_watcher/mod.rs @@ -0,0 +1,148 @@ +use anyhow::{Context as _, Result}; +use log::{debug, warn}; +use notify::{Config, Event, EventKind, RecommendedWatcher, RecursiveMode, Watcher as _}; +use std::path::Path; +use std::sync::mpsc::{self, Receiver}; +use std::time::Duration; + +pub use self::position_tracker::PositionTracker; + +mod position_tracker; + +/// Events that can occur during file watching +#[derive(Debug, Clone)] + +pub enum FileEvent { + /// New data has been written to the file + DataWritten, + + /// File was moved or renamed + Moved, + /// File was deleted + Deleted, + /// File was created (for handling log rotation) + Created, +} + +/// Cross-platform file watcher for monitoring log files +pub struct LogFileWatcher { + _watcher: RecommendedWatcher, + receiver: Receiver>, + file_path: std::path::PathBuf, +} + +impl LogFileWatcher { + /// Create a new file watcher for the specified file + pub fn new>(file_path: P) -> Result { + let file_path = file_path.as_ref().to_path_buf(); + + let (tx, receiver) = mpsc::channel(); + + // Configure watcher with appropriate settings for log monitoring + let config = Config::default() + .with_poll_interval(Duration::from_millis(500)) + .with_compare_contents(false); // We only care about size changes + + let mut watcher = RecommendedWatcher::new( + move |res| { + if let Err(e) = tx.send(res) { + warn!("Failed to send file system event: {}", e); + } + }, + config, + )?; + + // Watch the file directly if possible, otherwise watch its parent directory + let watch_path = if file_path.is_file() { + &file_path + } else if let Some(parent) = file_path.parent() { + parent + } else { + &file_path + }; + + watcher + .watch(watch_path, RecursiveMode::NonRecursive) + .with_context(|| format!("Failed to watch path: {}", watch_path.display()))?; + + debug!("Started watching file: {}", file_path.display()); + + Ok(LogFileWatcher { + _watcher: watcher, + receiver, + file_path, + }) + } + + /// Check for file events with a timeout + /// Returns None if no events occur within the timeout period + pub fn check_events(&self, timeout: Duration) -> Result> { + match self.receiver.recv_timeout(timeout) { + Ok(Ok(event)) => { + debug!("File system event: {:?}", event); + Ok(self.process_event(event)) + } + Ok(Err(e)) => { + warn!("File system notification error: {}", e); + Ok(None) + } + Err(mpsc::RecvTimeoutError::Timeout) => Ok(None), + Err(mpsc::RecvTimeoutError::Disconnected) => { + anyhow::bail!("File watcher channel disconnected"); + } + } + } + + /// Process a file system event and convert it to a LogEvent + fn process_event(&self, event: Event) -> Option { + // Only process events for our target file + if !event.paths.iter().any(|p| p == &self.file_path) { + return None; + } + + match event.kind { + EventKind::Modify(notify::event::ModifyKind::Data(_)) => { + // File content was modified + if let Ok(_metadata) = std::fs::metadata(&self.file_path) { + Some(FileEvent::DataWritten) + } else { + // File might have been deleted + Some(FileEvent::Deleted) + } + } + EventKind::Modify(notify::event::ModifyKind::Metadata(_)) => { + // File metadata changed, check if size changed + if let Ok(_metadata) = std::fs::metadata(&self.file_path) { + Some(FileEvent::DataWritten) + } else { + None + } + } + EventKind::Remove(_) => Some(FileEvent::Deleted), + EventKind::Create(_) => Some(FileEvent::Created), + EventKind::Modify(notify::event::ModifyKind::Name(notify::event::RenameMode::To)) => { + Some(FileEvent::Moved) + } + _ => { + debug!("Ignoring file system event: {:?}", event.kind); + None + } + } + } +} + +/// Handle graceful shutdown signals +pub fn setup_signal_handlers() -> Result> { + let (tx, rx) = mpsc::channel(); + + // Handle SIGINT (Ctrl+C) and SIGTERM + ctrlc::set_handler(move || { + debug!("Received shutdown signal"); + if let Err(e) = tx.send(()) { + warn!("Failed to send shutdown signal: {}", e); + } + }) + .context("Failed to set signal handler")?; + + Ok(rx) +} diff --git a/src/utils/file_watcher/position_tracker.rs b/src/utils/file_watcher/position_tracker.rs new file mode 100644 index 0000000000..593e727515 --- /dev/null +++ b/src/utils/file_watcher/position_tracker.rs @@ -0,0 +1,225 @@ +#![allow(clippy::allow_attributes)] + +use anyhow::{Context as _, Result}; +use log::debug; +use std::fs::File; +use std::io::{BufRead as _, BufReader, Seek as _, SeekFrom}; +use std::path::Path; + +/// Tracks file position for tail-like behavior +/// Handles file rotation and growth scenarios +#[derive(Debug)] +pub struct PositionTracker { + file_path: std::path::PathBuf, + current_position: u64, + current_size: u64, + inode: Option, +} + +impl PositionTracker { + /// Create a new position tracker starting from the end of the file + pub fn new>(file_path: P) -> Result { + let file_path = file_path.as_ref().to_path_buf(); + let metadata = std::fs::metadata(&file_path) + .with_context(|| format!("Failed to get metadata for {}", file_path.display()))?; + + let current_size = metadata.len(); + let inode = get_inode(&metadata); + + debug!( + "Initialized position tracker for {} at position {} (size: {})", + file_path.display(), + current_size, + current_size + ); + + Ok(PositionTracker { + file_path, + current_position: current_size, + current_size, + inode, + }) + } + + /// Check if the file has new data since the last read + /// Returns the number of new bytes available + pub fn check_for_new_data(&mut self) -> Result { + let metadata = std::fs::metadata(&self.file_path) + .with_context(|| format!("Failed to get metadata for {}", self.file_path.display()))?; + + let new_size = metadata.len(); + let new_inode = get_inode(&metadata); + + // Check if file was rotated (inode changed) + if self.inode.is_some() && new_inode != self.inode { + debug!("File rotation detected for {}", self.file_path.display()); + self.handle_file_rotation(new_size, new_inode); + return Ok(0); // No new data from current position after rotation + } + + // Check if file was truncated + if new_size < self.current_size { + debug!( + "File truncation detected: {} -> {} bytes", + self.current_size, new_size + ); + self.current_position = 0; + self.current_size = new_size; + self.inode = new_inode; + return Ok(new_size); + } + + // Normal case: file grew + let new_bytes = new_size.saturating_sub(self.current_size); + self.current_size = new_size; + self.inode = new_inode; + + Ok(new_bytes) + } + + /// Read new lines from the file since the last position + pub fn read_new_lines(&mut self) -> Result> { + let mut file = File::open(&self.file_path) + .with_context(|| format!("Failed to open {}", self.file_path.display()))?; + + // Seek to our current position + file.seek(SeekFrom::Start(self.current_position)) + .context("Failed to seek to current position")?; + + let mut reader = BufReader::new(file); + let mut lines = Vec::new(); + let mut line = String::new(); + + // Read lines from current position to end of file + loop { + line.clear(); + let bytes_read = reader + .read_line(&mut line) + .context("Failed to read line from file")?; + + if bytes_read == 0 { + break; // End of file + } + + // Remove trailing newline + if line.ends_with('\n') { + line.pop(); + if line.ends_with('\r') { + line.pop(); + } + } + + if !line.is_empty() { + lines.push(line.clone()); + } + + self.current_position += bytes_read as u64; + } + + debug!( + "Read {} new lines from {}", + lines.len(), + self.file_path.display() + ); + + Ok(lines) + } + + /// Handle file rotation scenario + fn handle_file_rotation(&mut self, new_size: u64, new_inode: Option) { + debug!("Handling file rotation, resetting position to 0"); + self.current_position = 0; + self.current_size = new_size; + self.inode = new_inode; + } + + /// Get the current file position + #[allow(dead_code)] + pub fn current_position(&self) -> u64 { + self.current_position + } + + /// Get the current file size + #[allow(dead_code)] + pub fn current_size(&self) -> u64 { + self.current_size + } +} + +/// Get file inode number for rotation detection (Unix-like systems) +#[cfg(unix)] +fn get_inode(metadata: &std::fs::Metadata) -> Option { + use std::os::unix::fs::MetadataExt as _; + // Unix systems always have inodes, but we return Option for API consistency + if metadata.len() == 0 && metadata.ino() == 0 { + // Handle edge case of empty/invalid file + None + } else { + Some(metadata.ino()) + } +} + +/// Windows doesn't have inodes, so we use creation time as a substitute +/// This helps detect file rotation when a new file is created with the same name +#[cfg(windows)] +#[allow(clippy::unnecessary_wraps)] // Option is needed for API consistency across platforms +fn get_inode(metadata: &std::fs::Metadata) -> Option { + use std::os::windows::fs::MetadataExt as _; + // Use creation time as a substitute for inode + // This is a stable API and works for detecting file rotation + Some(metadata.creation_time()) +} + +/// For other platforms, we can't detect rotation reliably +#[cfg(not(any(unix, windows)))] +fn get_inode(_metadata: &std::fs::Metadata) -> Option { + None +} + +#[cfg(test)] +mod tests { + use super::*; + use std::io::Write as _; + use tempfile::NamedTempFile; + + #[test] + fn test_position_tracker_new_file() -> Result<()> { + let mut temp_file = NamedTempFile::new()?; + writeln!(temp_file, "initial line")?; + temp_file.flush()?; + + let tracker = PositionTracker::new(temp_file.path())?; + + // Should start at end of file + assert!(tracker.current_position() > 0); + assert_eq!(tracker.current_position(), tracker.current_size()); + + Ok(()) + } + + #[test] + fn test_read_new_lines() -> Result<()> { + let mut temp_file = NamedTempFile::new()?; + writeln!(temp_file, "line 1")?; + temp_file.flush()?; + + let mut tracker = PositionTracker::new(temp_file.path())?; + + // Add new content + writeln!(temp_file, "line 2")?; + writeln!(temp_file, "line 3")?; + temp_file.flush()?; + + // Check for new data + let new_bytes = tracker.check_for_new_data()?; + assert!(new_bytes > 0); + + // Read new lines + let lines = tracker.read_new_lines()?; + assert_eq!(lines.len(), 2); + assert_eq!(lines[0], "line 2"); + assert_eq!(lines[1], "line 3"); + + Ok(()) + } +} diff --git a/src/utils/log_parsing/apache.rs b/src/utils/log_parsing/apache.rs new file mode 100644 index 0000000000..5ee0d568ac --- /dev/null +++ b/src/utils/log_parsing/apache.rs @@ -0,0 +1,399 @@ +use super::{LogEntry, LogFormat, LogLevel, LogParser}; +use anyhow::Result; +use chrono::{DateTime, NaiveDateTime, Utc}; +use regex::Regex; +use std::collections::HashMap; + +/// Parser for Apache Common Log Format (CLF) and Combined Log Format +pub struct ApacheParser { + /// Regex for Apache Common Log Format + common_regex: Regex, + /// Regex for Apache Combined Log Format (includes referer and user agent) + combined_regex: Regex, + /// Regex for Apache error logs + error_regex: Regex, +} + +impl ApacheParser { + pub fn new() -> Self { + // Apache Common Log Format: host ident authuser [timestamp] "request" status bytes + // Example: 127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 + let common_pattern = r#"^(\S+) (\S+) (\S+) \[([^\]]+)\] "([^"]*)" (\d+) (\S+)$"#; + let common_regex = Regex::new(common_pattern).expect("Invalid Apache common regex"); + + // Apache Combined Log Format: common format + "referer" "user_agent" + // Example: 127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "http://www.example.com/start.html" "Mozilla/4.08" + let combined_pattern = + r#"^(\S+) (\S+) (\S+) \[([^\]]+)\] "([^"]*)" (\d+) (\S+) "([^"]*)" "([^"]*)""#; + let combined_regex = Regex::new(combined_pattern).expect("Invalid Apache combined regex"); + + // Apache error log format + // Example: [Wed Oct 11 14:32:52 2000] [error] [client 127.0.0.1] client denied by server configuration: /export/home/live/ap/htdocs/test + let error_pattern = r#"^\[([^\]]+)\] \[(\w+)\] (?:\[client ([^\]]+)\] )?(.+)$"#; + let error_regex = Regex::new(error_pattern).expect("Invalid Apache error regex"); + + ApacheParser { + common_regex, + combined_regex, + error_regex, + } + } + + /// Parse Apache Common Log Format + fn parse_common_log(&self, line: &str) -> Result { + if let Some(captures) = self.common_regex.captures(line) { + let mut fields = HashMap::new(); + + // Extract fields + let host = captures + .get(1) + .expect("regex capture group should exist") + .as_str(); + let ident = captures + .get(2) + .expect("regex capture group should exist") + .as_str(); + let authuser = captures + .get(3) + .expect("regex capture group should exist") + .as_str(); + let timestamp_str = captures + .get(4) + .expect("regex capture group should exist") + .as_str(); + let request = captures + .get(5) + .expect("regex capture group should exist") + .as_str(); + let status = captures + .get(6) + .expect("regex capture group should exist") + .as_str(); + let bytes = captures + .get(7) + .expect("regex capture group should exist") + .as_str(); + + fields.insert("remote_host".to_owned(), host.to_owned()); + if ident != "-" { + fields.insert("remote_ident".to_owned(), ident.to_owned()); + } + if authuser != "-" { + fields.insert("remote_user".to_owned(), authuser.to_owned()); + } + fields.insert("request".to_owned(), request.to_owned()); + fields.insert("status".to_owned(), status.to_owned()); + if bytes != "-" { + fields.insert("bytes_sent".to_owned(), bytes.to_owned()); + } + + // Parse timestamp - Apache format: 10/Oct/2000:13:55:36 -0700 + let timestamp = parse_apache_timestamp(timestamp_str); + + // Determine log level based on HTTP status + let level = match status.parse::().unwrap_or(200) { + 400..=499 => Some(LogLevel::Warning), + 500..=599 => Some(LogLevel::Error), + _ => Some(LogLevel::Info), + }; + + Ok(LogEntry { + message: line.to_owned(), + timestamp, + level, + fields, + format: LogFormat::Apache, + }) + } else { + anyhow::bail!("Failed to parse Apache common log line: {}", line); + } + } + + /// Parse Apache Combined Log Format + fn parse_combined_log(&self, line: &str) -> Result { + if let Some(captures) = self.combined_regex.captures(line) { + let mut fields = HashMap::new(); + + // Extract all fields from combined format + let host = captures + .get(1) + .expect("regex capture group should exist") + .as_str(); + let ident = captures + .get(2) + .expect("regex capture group should exist") + .as_str(); + let authuser = captures + .get(3) + .expect("regex capture group should exist") + .as_str(); + let timestamp_str = captures + .get(4) + .expect("regex capture group should exist") + .as_str(); + let request = captures + .get(5) + .expect("regex capture group should exist") + .as_str(); + let status = captures + .get(6) + .expect("regex capture group should exist") + .as_str(); + let bytes = captures + .get(7) + .expect("regex capture group should exist") + .as_str(); + let referer = captures + .get(8) + .expect("regex capture group should exist") + .as_str(); + let user_agent = captures + .get(9) + .expect("regex capture group should exist") + .as_str(); + + fields.insert("remote_host".to_owned(), host.to_owned()); + if ident != "-" { + fields.insert("remote_ident".to_owned(), ident.to_owned()); + } + if authuser != "-" { + fields.insert("remote_user".to_owned(), authuser.to_owned()); + } + fields.insert("request".to_owned(), request.to_owned()); + fields.insert("status".to_owned(), status.to_owned()); + if bytes != "-" { + fields.insert("bytes_sent".to_owned(), bytes.to_owned()); + } + if referer != "-" { + fields.insert("http_referer".to_owned(), referer.to_owned()); + } + if user_agent != "-" { + fields.insert("http_user_agent".to_owned(), user_agent.to_owned()); + } + + let timestamp = parse_apache_timestamp(timestamp_str); + + let level = match status.parse::().unwrap_or(200) { + 400..=499 => Some(LogLevel::Warning), + 500..=599 => Some(LogLevel::Error), + _ => Some(LogLevel::Info), + }; + + Ok(LogEntry { + message: line.to_owned(), + timestamp, + level, + fields, + format: LogFormat::Apache, + }) + } else { + anyhow::bail!("Failed to parse Apache combined log line: {}", line); + } + } + + /// Parse Apache error log format + fn parse_error_log(&self, line: &str) -> Result { + if let Some(captures) = self.error_regex.captures(line) { + let mut fields = HashMap::new(); + + let timestamp_str = captures + .get(1) + .expect("regex capture group should exist") + .as_str(); + let level_str = captures + .get(2) + .expect("regex capture group should exist") + .as_str(); + let client = captures.get(3).map(|m| m.as_str()); + let message = captures + .get(4) + .expect("regex capture group should exist") + .as_str(); + + if let Some(client_ip) = client { + fields.insert("client_ip".to_owned(), client_ip.to_owned()); + } + fields.insert("error_message".to_owned(), message.to_owned()); + + // Parse timestamp - Apache error format: Wed Oct 11 14:32:52 2000 + let timestamp = parse_apache_error_timestamp(timestamp_str); + + // Parse log level + let level = LogLevel::from_str(level_str).or({ + // Apache specific levels + match level_str { + "emerg" | "alert" | "crit" => Some(LogLevel::Fatal), + "error" => Some(LogLevel::Error), + "warn" => Some(LogLevel::Warning), + "notice" | "info" => Some(LogLevel::Info), + "debug" => Some(LogLevel::Debug), + _ => None, + } + }); + + Ok(LogEntry { + message: line.to_owned(), + timestamp, + level, + fields, + format: LogFormat::Apache, + }) + } else { + anyhow::bail!("Failed to parse Apache error log line: {}", line); + } + } +} + +impl LogParser for ApacheParser { + fn parse_line(&self, line: &str) -> Result { + // Try combined format first (more specific), then common, then error + if self.combined_regex.is_match(line) { + self.parse_combined_log(line) + } else if self.common_regex.is_match(line) { + self.parse_common_log(line) + } else if self.error_regex.is_match(line) { + self.parse_error_log(line) + } else { + anyhow::bail!("Line does not match Apache log format: {}", line); + } + } + + fn can_parse(&self, line: &str) -> bool { + self.combined_regex.is_match(line) + || self.common_regex.is_match(line) + || self.error_regex.is_match(line) + } + + fn format(&self) -> LogFormat { + LogFormat::Apache + } +} + +/// Parse Apache access log timestamp: 10/Oct/2000:13:55:36 -0700 +fn parse_apache_timestamp(timestamp_str: &str) -> Option> { + // Remove timezone part for parsing with chrono + let without_tz = timestamp_str.split(' ').next()?; + + // Parse the datetime part + NaiveDateTime::parse_from_str(without_tz, "%d/%b/%Y:%H:%M:%S") + .ok() + .map(|dt| DateTime::from_naive_utc_and_offset(dt, Utc)) +} + +/// Parse Apache error log timestamp: Wed Oct 11 14:32:52 2000 +fn parse_apache_error_timestamp(timestamp_str: &str) -> Option> { + NaiveDateTime::parse_from_str(timestamp_str, "%a %b %d %H:%M:%S %Y") + .ok() + .map(|dt| DateTime::from_naive_utc_and_offset(dt, Utc)) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_parse_apache_common_log() { + let parser = ApacheParser::new(); + let line = r#"127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326"#; + + let entry = parser + .parse_line(line) + .expect("regex capture group should exist"); + assert_eq!(entry.message, line); + assert!(entry.timestamp.is_some()); + assert!(matches!(entry.level, Some(LogLevel::Info))); + assert_eq!( + entry + .fields + .get("remote_host") + .expect("regex capture group should exist"), + "127.0.0.1" + ); + assert_eq!( + entry + .fields + .get("remote_user") + .expect("regex capture group should exist"), + "frank" + ); + assert_eq!( + entry + .fields + .get("status") + .expect("regex capture group should exist"), + "200" + ); + assert!(matches!(entry.format, LogFormat::Apache)); + } + + #[test] + fn test_parse_apache_combined_log() { + let parser = ApacheParser::new(); + let line = r#"127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "http://www.example.com/start.html" "Mozilla/4.08""#; + + let entry = parser + .parse_line(line) + .expect("regex capture group should exist"); + assert_eq!(entry.message, line); + assert!(entry.timestamp.is_some()); + assert!(matches!(entry.level, Some(LogLevel::Info))); + assert_eq!( + entry + .fields + .get("remote_host") + .expect("regex capture group should exist"), + "127.0.0.1" + ); + assert_eq!( + entry + .fields + .get("http_referer") + .expect("regex capture group should exist"), + "http://www.example.com/start.html" + ); + assert_eq!( + entry + .fields + .get("http_user_agent") + .expect("regex capture group should exist"), + "Mozilla/4.08" + ); + assert!(matches!(entry.format, LogFormat::Apache)); + } + + #[test] + fn test_parse_apache_error_log() { + let parser = ApacheParser::new(); + let line = "[Wed Oct 11 14:32:52 2000] [error] [client 127.0.0.1] client denied by server configuration"; + + let entry = parser + .parse_line(line) + .expect("regex capture group should exist"); + assert_eq!(entry.message, line); + assert!(entry.timestamp.is_some()); + assert!(matches!(entry.level, Some(LogLevel::Error))); + assert_eq!( + entry + .fields + .get("client_ip") + .expect("regex capture group should exist"), + "127.0.0.1" + ); + assert!(entry.fields.contains_key("error_message")); + assert!(matches!(entry.format, LogFormat::Apache)); + } + + #[test] + fn test_can_parse_apache_logs() { + let parser = ApacheParser::new(); + + let common_line = r#"127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326"#; + assert!(parser.can_parse(common_line)); + + let error_line = "[Wed Oct 11 14:32:52 2000] [error] client denied by server configuration"; + assert!(parser.can_parse(error_line)); + + let invalid_line = "This is not an Apache log"; + assert!(!parser.can_parse(invalid_line)); + } +} diff --git a/src/utils/log_parsing/mod.rs b/src/utils/log_parsing/mod.rs new file mode 100644 index 0000000000..b880d6b6dc --- /dev/null +++ b/src/utils/log_parsing/mod.rs @@ -0,0 +1,157 @@ +use anyhow::Result; +use chrono::{DateTime, Utc}; +use serde::{Deserialize, Serialize}; +use std::collections::HashMap; + +pub mod apache; +pub mod nginx; +pub mod plain; + +/// A structured log entry that can be sent to Sentry +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct LogEntry { + /// The original raw log line + pub message: String, + /// Parsed timestamp (if available) + pub timestamp: Option>, + /// Log level/severity (if detected) + pub level: Option, + /// Additional structured fields extracted from the log + pub fields: HashMap, + /// The format that was used to parse this entry + pub format: LogFormat, +} + +/// Log severity levels that map to Sentry levels +#[derive(Debug, Clone, Serialize, Deserialize)] +pub enum LogLevel { + Debug, + Info, + Warning, + Error, + Fatal, +} + +impl LogLevel { + /// Convert to Sentry level string + pub fn to_sentry_level(&self) -> &'static str { + match self { + LogLevel::Debug => "debug", + LogLevel::Info => "info", + LogLevel::Warning => "warning", + LogLevel::Error => "error", + LogLevel::Fatal => "fatal", + } + } + + /// Parse log level from string (case insensitive) + pub fn from_str(s: &str) -> Option { + match s.to_lowercase().as_str() { + "debug" | "dbg" => Some(LogLevel::Debug), + "info" | "information" => Some(LogLevel::Info), + "warn" | "warning" => Some(LogLevel::Warning), + "error" | "err" => Some(LogLevel::Error), + "fatal" | "critical" | "crit" => Some(LogLevel::Fatal), + _ => None, + } + } +} + +/// Supported log formats +#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq, Hash)] +pub enum LogFormat { + Nginx, + Apache, + Plain, +} + +impl LogFormat { + pub fn name(&self) -> &'static str { + match self { + LogFormat::Nginx => "nginx", + LogFormat::Apache => "apache", + LogFormat::Plain => "plain", + } + } +} + +/// Trait for log format parsers +pub trait LogParser { + /// Parse a single log line into a structured LogEntry + fn parse_line(&self, line: &str) -> Result; + + /// Check if this parser can handle the given log line + /// Used for auto-detection + fn can_parse(&self, line: &str) -> bool; + + /// Get the format name + fn format(&self) -> LogFormat; +} + +/// Auto-detect the log format based on sample lines +pub fn detect_log_format(sample_lines: &[String]) -> LogFormat { + let parsers: Vec> = vec![ + Box::new(nginx::NginxParser::new()), + Box::new(apache::ApacheParser::new()), + ]; + + // Count successful parses for each format + let mut format_scores: HashMap = HashMap::new(); + + for line in sample_lines.iter().take(10) { + // Check first 10 lines + if line.trim().is_empty() { + continue; + } + + for parser in &parsers { + if parser.can_parse(line) { + *format_scores.entry(parser.format()).or_insert(0) += 1; + } + } + } + + // Return the format with the highest score, or Plain as fallback + format_scores + .into_iter() + .max_by_key(|(_, score)| *score) + .map(|(format, _)| format) + .unwrap_or(LogFormat::Plain) +} + +/// Create a parser for the specified format +pub fn create_parser(format: LogFormat) -> Box { + match format { + LogFormat::Nginx => Box::new(nginx::NginxParser::new()), + LogFormat::Apache => Box::new(apache::ApacheParser::new()), + LogFormat::Plain => Box::new(plain::PlainParser::new()), + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_log_level_from_str() { + assert_eq!( + LogLevel::from_str("ERROR").unwrap().to_sentry_level(), + "error" + ); + assert_eq!( + LogLevel::from_str("info").unwrap().to_sentry_level(), + "info" + ); + assert_eq!( + LogLevel::from_str("WARN").unwrap().to_sentry_level(), + "warning" + ); + assert!(LogLevel::from_str("invalid").is_none()); + } + + #[test] + fn test_detect_log_format_fallback() { + let sample_lines = vec!["some random text".to_owned()]; + assert!(matches!(detect_log_format(&sample_lines), LogFormat::Plain)); + } +} diff --git a/src/utils/log_parsing/nginx.rs b/src/utils/log_parsing/nginx.rs new file mode 100644 index 0000000000..010973d364 --- /dev/null +++ b/src/utils/log_parsing/nginx.rs @@ -0,0 +1,253 @@ +use super::{LogEntry, LogFormat, LogLevel, LogParser}; +use anyhow::Result; +use chrono::{DateTime, NaiveDateTime, Utc}; +use lazy_static::lazy_static; +use regex::Regex; +use std::collections::HashMap; + +lazy_static! { + /// Compiled regex for nginx combined log format + /// Example: 192.168.1.1 - - [25/Dec/2023:10:00:00 +0000] "GET /index.html HTTP/1.1" 200 1024 "http://example.com" "Mozilla/5.0" + static ref NGINX_COMBINED_REGEX: Regex = Regex::new( + r#"^(\S+) \S+ \S+ \[([^\]]+)\] "([^"]*)" (\d+) (\S+) "([^"]*)" "([^"]*)""# + ).expect("Invalid nginx combined regex"); + + /// Compiled regex for nginx error log format + /// Example: 2023/12/25 10:00:00 [error] 1234#0: *1 connect() failed (111: Connection refused) + static ref NGINX_ERROR_REGEX: Regex = Regex::new( + r#"^(\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2}) \[(\w+)\] \d+#\d+: (.+)$"# + ).expect("Invalid nginx error regex"); +} + +/// Parser for nginx default log format +/// Default format: '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"' +pub struct NginxParser; + +impl NginxParser { + pub fn new() -> Self { + NginxParser + } + + /// Parse nginx combined/access log format + fn parse_combined_log(&self, line: &str) -> Result { + if let Some(captures) = NGINX_COMBINED_REGEX.captures(line) { + let mut fields = HashMap::new(); + + // Extract fields + fields.insert( + "remote_addr".to_owned(), + captures + .get(1) + .expect("regex capture group should exist") + .as_str() + .to_owned(), + ); + + let timestamp_str = captures + .get(2) + .expect("regex capture group should exist") + .as_str(); + let request = captures + .get(3) + .expect("regex capture group should exist") + .as_str(); + let status = captures + .get(4) + .expect("regex capture group should exist") + .as_str(); + let body_bytes = captures + .get(5) + .expect("regex capture group should exist") + .as_str(); + let referer = captures + .get(6) + .expect("regex capture group should exist") + .as_str(); + let user_agent = captures + .get(7) + .expect("regex capture group should exist") + .as_str(); + + fields.insert("request".to_owned(), request.to_owned()); + fields.insert("status".to_owned(), status.to_owned()); + fields.insert("body_bytes_sent".to_owned(), body_bytes.to_owned()); + + if referer != "-" { + fields.insert("http_referer".to_owned(), referer.to_owned()); + } + if user_agent != "-" { + fields.insert("http_user_agent".to_owned(), user_agent.to_owned()); + } + + // Parse timestamp - nginx format: 25/Dec/2023:10:00:00 +0000 + let timestamp = parse_nginx_timestamp(timestamp_str); + + // Determine log level based on HTTP status + let level = match status.parse::().unwrap_or(200) { + 400..=499 => Some(LogLevel::Warning), + 500..=599 => Some(LogLevel::Error), + _ => Some(LogLevel::Info), + }; + + Ok(LogEntry { + message: line.to_owned(), + timestamp, + level, + fields, + format: LogFormat::Nginx, + }) + } else { + anyhow::bail!("Failed to parse nginx combined log line: {}", line); + } + } + + /// Parse nginx error log format + fn parse_error_log(&self, line: &str) -> Result { + if let Some(captures) = NGINX_ERROR_REGEX.captures(line) { + let mut fields = HashMap::new(); + + let timestamp_str = captures + .get(1) + .expect("regex capture group should exist") + .as_str(); + let level_str = captures + .get(2) + .expect("regex capture group should exist") + .as_str(); + let message = captures + .get(3) + .expect("regex capture group should exist") + .as_str(); + + fields.insert("error_message".to_owned(), message.to_owned()); + + // Parse timestamp - nginx error format: 2023/12/25 10:00:00 + let timestamp = parse_nginx_error_timestamp(timestamp_str); + + // Parse log level + let level = LogLevel::from_str(level_str).or({ + // nginx specific levels + match level_str { + "emerg" | "alert" | "crit" => Some(LogLevel::Fatal), + "err" => Some(LogLevel::Error), + "warn" => Some(LogLevel::Warning), + "notice" | "info" => Some(LogLevel::Info), + "debug" => Some(LogLevel::Debug), + _ => None, + } + }); + + Ok(LogEntry { + message: line.to_owned(), + timestamp, + level, + fields, + format: LogFormat::Nginx, + }) + } else { + anyhow::bail!("Failed to parse nginx error log line: {}", line); + } + } +} + +impl LogParser for NginxParser { + fn parse_line(&self, line: &str) -> Result { + // Try combined format first, then error format + if NGINX_COMBINED_REGEX.is_match(line) { + self.parse_combined_log(line) + } else if NGINX_ERROR_REGEX.is_match(line) { + self.parse_error_log(line) + } else { + anyhow::bail!("Line does not match nginx log format: {}", line); + } + } + + fn can_parse(&self, line: &str) -> bool { + NGINX_COMBINED_REGEX.is_match(line) || NGINX_ERROR_REGEX.is_match(line) + } + + fn format(&self) -> LogFormat { + LogFormat::Nginx + } +} + +/// Parse nginx access log timestamp: 25/Dec/2023:10:00:00 +0000 +fn parse_nginx_timestamp(timestamp_str: &str) -> Option> { + // Remove timezone part for parsing with chrono + let without_tz = timestamp_str.split(' ').next()?; + + // Parse the datetime part + NaiveDateTime::parse_from_str(without_tz, "%d/%b/%Y:%H:%M:%S") + .ok() + .map(|dt| DateTime::from_naive_utc_and_offset(dt, Utc)) +} + +/// Parse nginx error log timestamp: 2023/12/25 10:00:00 +fn parse_nginx_error_timestamp(timestamp_str: &str) -> Option> { + NaiveDateTime::parse_from_str(timestamp_str, "%Y/%m/%d %H:%M:%S") + .ok() + .map(|dt| DateTime::from_naive_utc_and_offset(dt, Utc)) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_parse_nginx_combined_log() { + let parser = NginxParser::new(); + let line = r#"192.168.1.1 - - [25/Dec/2023:10:00:00 +0000] "GET /index.html HTTP/1.1" 200 1024 "http://example.com" "Mozilla/5.0""#; + + let entry = parser + .parse_line(line) + .expect("regex capture group should exist"); + assert_eq!(entry.message, line); + assert!(entry.timestamp.is_some()); + assert!(matches!(entry.level, Some(LogLevel::Info))); + assert_eq!( + entry + .fields + .get("remote_addr") + .expect("regex capture group should exist"), + "192.168.1.1" + ); + assert_eq!( + entry + .fields + .get("status") + .expect("regex capture group should exist"), + "200" + ); + assert!(matches!(entry.format, LogFormat::Nginx)); + } + + #[test] + fn test_parse_nginx_error_log() { + let parser = NginxParser::new(); + let line = + "2023/12/25 10:00:00 [error] 1234#0: *1 connect() failed (111: Connection refused)"; + + let entry = parser + .parse_line(line) + .expect("regex capture group should exist"); + assert_eq!(entry.message, line); + assert!(entry.timestamp.is_some()); + assert!(matches!(entry.level, Some(LogLevel::Error))); + assert!(entry.fields.contains_key("error_message")); + assert!(matches!(entry.format, LogFormat::Nginx)); + } + + #[test] + fn test_can_parse_nginx_logs() { + let parser = NginxParser::new(); + + let combined_line = r#"192.168.1.1 - - [25/Dec/2023:10:00:00 +0000] "GET /index.html HTTP/1.1" 200 1024 "http://example.com" "Mozilla/5.0""#; + assert!(parser.can_parse(combined_line)); + + let error_line = "2023/12/25 10:00:00 [error] 1234#0: *1 connect() failed"; + assert!(parser.can_parse(error_line)); + + let invalid_line = "This is not an nginx log"; + assert!(!parser.can_parse(invalid_line)); + } +} diff --git a/src/utils/log_parsing/plain.rs b/src/utils/log_parsing/plain.rs new file mode 100644 index 0000000000..b41cbd2804 --- /dev/null +++ b/src/utils/log_parsing/plain.rs @@ -0,0 +1,277 @@ +use super::{LogEntry, LogFormat, LogLevel, LogParser}; +use anyhow::Result; +use chrono::{DateTime, Utc}; +use regex::Regex; +use std::collections::HashMap; + +/// Simple parser for plain text logs +/// Attempts to extract timestamps and log levels from unstructured text +pub struct PlainParser { + /// Regex for detecting common timestamp patterns + timestamp_regex: Regex, + /// Regex for detecting log levels in text + level_regex: Regex, +} + +impl PlainParser { + pub fn new() -> Self { + // Common timestamp patterns + // Matches: 2023-12-25 10:00:00, 2023/12/25 10:00:00, Dec 25 10:00:00, etc. + let timestamp_pattern = r"(\d{4}[-/]\d{2}[-/]\d{2}[T\s]\d{2}:\d{2}:\d{2}|\w{3}\s+\d{1,2}\s+\d{2}:\d{2}:\d{2}|\d{2}/\w{3}/\d{4}:\d{2}:\d{2}:\d{2})"; + let timestamp_regex = Regex::new(timestamp_pattern).expect("Invalid timestamp regex"); + + // Log level detection - case insensitive + let level_pattern = r"(?i)\b(TRACE|DEBUG|INFO|INFORMATION|WARN|WARNING|ERROR|ERR|FATAL|CRITICAL|CRIT|PANIC)\b"; + let level_regex = Regex::new(level_pattern).expect("Invalid level regex"); + + PlainParser { + timestamp_regex, + level_regex, + } + } + + /// Extract timestamp from plain text + fn extract_timestamp(&self, line: &str) -> Option> { + if let Some(timestamp_match) = self.timestamp_regex.find(line) { + let timestamp_str = timestamp_match.as_str(); + + // Try different timestamp formats + self.parse_timestamp_formats(timestamp_str) + } else { + None + } + } + + /// Try parsing various timestamp formats + fn parse_timestamp_formats(&self, timestamp_str: &str) -> Option> { + use chrono::NaiveDateTime; + + // List of common timestamp formats to try + let formats = vec![ + "%Y-%m-%d %H:%M:%S", // 2023-12-25 10:00:00 + "%Y/%m/%d %H:%M:%S", // 2023/12/25 10:00:00 + "%Y-%m-%dT%H:%M:%S", // 2023-12-25T10:00:00 + "%d/%b/%Y:%H:%M:%S", // 25/Dec/2023:10:00:00 + "%b %d %H:%M:%S", // Dec 25 10:00:00 (current year assumed) + ]; + + for format in formats { + if let Ok(dt) = NaiveDateTime::parse_from_str(timestamp_str, format) { + return Some(DateTime::from_naive_utc_and_offset(dt, Utc)); + } + } + + // Try with current year for formats without year + use chrono::Datelike as _; + if let Ok(dt) = chrono::NaiveDateTime::parse_from_str( + &format!("{} {timestamp_str}", chrono::Utc::now().year()), + "%Y %b %d %H:%M:%S", + ) { + return Some(DateTime::from_naive_utc_and_offset(dt, Utc)); + } + + None + } + + /// Extract log level from plain text + fn extract_level(&self, line: &str) -> Option { + if let Some(level_match) = self.level_regex.find(line) { + LogLevel::from_str(level_match.as_str()) + } else { + // Try to infer level from common keywords + let line_lower = line.to_lowercase(); + if line_lower.contains("exception") + || line_lower.contains("failed") + || line_lower.contains("error") + { + Some(LogLevel::Error) + } else if line_lower.contains("warning") || line_lower.contains("warn") { + Some(LogLevel::Warning) + } else { + // Default to info for plain text + Some(LogLevel::Info) + } + } + } + + /// Extract structured fields from plain text using common patterns + fn extract_fields(&self, line: &str) -> HashMap { + let mut fields = HashMap::new(); + + // Extract key=value pairs + let kv_regex = Regex::new(r"(\w+)=([^\s]+)").expect("Invalid KV regex"); + for captures in kv_regex.captures_iter(line) { + let key = captures + .get(1) + .expect("regex capture group should exist") + .as_str(); + let value = captures + .get(2) + .expect("regex capture group should exist") + .as_str(); + fields.insert(key.to_owned(), value.to_owned()); + } + + // Extract quoted strings that might be values + let quoted_regex = Regex::new(r#""([^"]+)""#).expect("Invalid quoted regex"); + let mut quoted_values = Vec::new(); + for captures in quoted_regex.captures_iter(line) { + if let Some(quoted) = captures.get(1) { + quoted_values.push(quoted.as_str().to_owned()); + } + } + + // Store quoted values if found + for (i, value) in quoted_values.iter().enumerate() { + fields.insert(format!("quoted_field_{i}"), value.clone()); + } + + fields + } +} + +impl LogParser for PlainParser { + fn parse_line(&self, line: &str) -> Result { + let timestamp = self.extract_timestamp(line); + let level = self.extract_level(line); + let fields = self.extract_fields(line); + + Ok(LogEntry { + message: line.to_owned(), + timestamp, + level, + fields, + format: LogFormat::Plain, + }) + } + + fn can_parse(&self, _line: &str) -> bool { + // Plain parser can handle any line as a fallback + true + } + + fn format(&self) -> LogFormat { + LogFormat::Plain + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_parse_plain_log_with_timestamp() { + let parser = PlainParser::new(); + let line = "2023-12-25 10:00:00 ERROR Something went wrong in the application"; + + let entry = parser + .parse_line(line) + .expect("regex capture group should exist"); + assert_eq!(entry.message, line); + assert!(entry.timestamp.is_some()); + assert!(matches!(entry.level, Some(LogLevel::Error))); + assert!(matches!(entry.format, LogFormat::Plain)); + } + + #[test] + fn test_parse_plain_log_with_key_value() { + let parser = PlainParser::new(); + let line = "User logged in successfully user_id=12345 session=abc123"; + + let entry = parser + .parse_line(line) + .expect("regex capture group should exist"); + assert_eq!(entry.message, line); + assert!(matches!(entry.level, Some(LogLevel::Info))); + assert_eq!( + entry + .fields + .get("user_id") + .expect("regex capture group should exist"), + "12345" + ); + assert_eq!( + entry + .fields + .get("session") + .expect("regex capture group should exist"), + "abc123" + ); + assert!(matches!(entry.format, LogFormat::Plain)); + } + + #[test] + fn test_parse_plain_log_with_quoted_values() { + let parser = PlainParser::new(); + let line = r#"Processing request "GET /api/users" from client"#; + + let entry = parser + .parse_line(line) + .expect("regex capture group should exist"); + assert_eq!(entry.message, line); + assert!(matches!(entry.level, Some(LogLevel::Info))); + assert_eq!( + entry + .fields + .get("quoted_field_0") + .expect("regex capture group should exist"), + "GET /api/users" + ); + assert!(matches!(entry.format, LogFormat::Plain)); + } + + #[test] + fn test_extract_timestamp() { + let parser = PlainParser::new(); + + assert!(parser + .extract_timestamp("2023-12-25 10:00:00 message") + .is_some()); + assert!(parser + .extract_timestamp("2023/12/25 10:00:00 message") + .is_some()); + assert!(parser + .extract_timestamp("Dec 25 10:00:00 message") + .is_some()); + assert!(parser.extract_timestamp("no timestamp here").is_none()); + } + + #[test] + fn test_extract_level() { + let parser = PlainParser::new(); + + assert!(matches!( + parser.extract_level("ERROR: something failed"), + Some(LogLevel::Error) + )); + assert!(matches!( + parser.extract_level("WARNING: be careful"), + Some(LogLevel::Warning) + )); + assert!(matches!( + parser.extract_level("INFO: normal operation"), + Some(LogLevel::Info) + )); + assert!(matches!( + parser.extract_level("DEBUG: detailed info"), + Some(LogLevel::Debug) + )); + assert!(matches!( + parser.extract_level("exception occurred"), + Some(LogLevel::Error) + )); + assert!(matches!( + parser.extract_level("plain message"), + Some(LogLevel::Info) + )); + } + + #[test] + fn test_can_parse_any_line() { + let parser = PlainParser::new(); + + assert!(parser.can_parse("Any line should be parseable")); + assert!(parser.can_parse("Even empty lines")); + assert!(parser.can_parse("")); + } +} diff --git a/src/utils/log_transmission.rs b/src/utils/log_transmission.rs new file mode 100644 index 0000000000..acb32c18c3 --- /dev/null +++ b/src/utils/log_transmission.rs @@ -0,0 +1,389 @@ +use anyhow::{Context as _, Result}; +use chrono::{DateTime, Utc}; +use log::debug; + +use sentry::types::Uuid; +use serde::{Deserialize, Serialize}; +use std::collections::HashMap; +use std::time::{SystemTime, UNIX_EPOCH}; + +use crate::api::envelopes_api::EnvelopesApi; +use crate::utils::log_parsing::{LogEntry, LogLevel}; + +/// Sentry log payload as per the official logs protocol +/// https://develop.sentry.dev/sdk/telemetry/logs/#log-envelope-item-payload +#[derive(Debug, Serialize, Deserialize)] +pub struct SentryLogPayload { + /// The timestamp of the log in seconds since the Unix epoch + pub timestamp: f64, + /// The trace id of the log (16 random bytes encoded as hex string) + pub trace_id: String, + /// The severity level of the log + pub level: String, + /// The log body/message + pub body: String, + /// Dictionary of key-value pairs with typed values + #[serde(skip_serializing_if = "HashMap::is_empty")] + pub attributes: HashMap, + /// Optional severity number + #[serde(skip_serializing_if = "Option::is_none")] + pub severity_number: Option, +} + +/// Attribute value with type information as per Sentry logs protocol +#[derive(Debug, Serialize, Deserialize)] +pub struct SentryLogAttribute { + pub value: serde_json::Value, + #[serde(rename = "type")] + pub attr_type: String, +} + +/// Envelope payload for log items as per Sentry specification +#[derive(Debug, Serialize, Deserialize)] +pub struct LogEnvelopeItems { + pub items: Vec, +} + +/// Converts our parsed log entries into proper Sentry log envelopes and sends them +pub struct LogTransmitter { + envelope_api: EnvelopesApi, + org: String, + project: String, +} + +impl LogTransmitter { + /// Create a new log transmitter + pub fn new(org: String, project: String) -> Result { + let envelope_api = EnvelopesApi::try_new() + .context("Failed to initialize Sentry envelope API. Check your DSN configuration.")?; + + Ok(LogTransmitter { + envelope_api, + org, + project, + }) + } + + /// Send a batch of log entries to Sentry using the proper logs protocol + pub fn send_log_batch(&self, log_entries: Vec) -> Result> { + if log_entries.is_empty() { + return Ok(Vec::new()); + } + + let mut sentry_logs = Vec::new(); + + for log_entry_json in log_entries { + let sentry_log = self.convert_to_sentry_log(&log_entry_json); + sentry_logs.push(sentry_log); + } + + if sentry_logs.is_empty() { + return Ok(Vec::new()); + } + + // Send all logs in a single envelope with proper specification format + // https://develop.sentry.dev/sdk/telemetry/logs/#appendix-a-example-log-envelope + let envelope_payload = LogEnvelopeItems { items: sentry_logs }; + + let envelope_json = serde_json::to_string(&envelope_payload) + .context("Failed to serialize log envelope payload")?; + + // Create proper log envelope according to specification + let envelope_header = "{}"; // Empty header for logs (no event_id needed) + let item_header = format!( + r#"{{"type":"log","item_count":{},"content_type":"application/vnd.sentry.items.log+json"}}"#, + envelope_payload.items.len() + ); + let envelope_content = format!("{envelope_header}\n{item_header}\n{envelope_json}"); + + debug!("Sending log envelope:\n{}", envelope_content); + + // Send the envelope using the raw API + self.send_raw_envelope(envelope_content.into_bytes()) + .context("Failed to send log envelope")?; + + debug!( + "Successfully sent {} log entries to Sentry", + envelope_payload.items.len() + ); + Ok(vec![Uuid::new_v4()]) // Return single ID for the envelope + } + + /// Convert a log entry JSON string to Sentry log format + fn convert_to_sentry_log(&self, log_entry_json: &str) -> SentryLogPayload { + // First, try to deserialize as our LogEntry type + match serde_json::from_str::(log_entry_json) { + Ok(log_entry) => self.create_structured_sentry_log(&log_entry), + Err(_) => { + // Fall back to treating it as plain text + self.create_plain_text_sentry_log(log_entry_json) + } + } + } + + /// Create a structured Sentry log from our LogEntry + fn create_structured_sentry_log(&self, log_entry: &LogEntry) -> SentryLogPayload { + let mut attributes = HashMap::new(); + + // Add structured fields as attributes with proper typing + for (key, value) in &log_entry.fields { + attributes.insert( + key.clone(), + SentryLogAttribute { + value: serde_json::Value::String(value.clone()), + attr_type: "string".to_owned(), + }, + ); + } + + // Add SDK information as per spec + attributes.insert( + "sentry.sdk.name".to_owned(), + SentryLogAttribute { + value: serde_json::Value::String("sentry-cli".to_owned()), + attr_type: "string".to_owned(), + }, + ); + attributes.insert( + "sentry.sdk.version".to_owned(), + SentryLogAttribute { + value: serde_json::Value::String(env!("CARGO_PKG_VERSION").to_owned()), + attr_type: "string".to_owned(), + }, + ); + + // Add format information + attributes.insert( + "log_format".to_owned(), + SentryLogAttribute { + value: serde_json::Value::String(log_entry.format.name().to_owned()), + attr_type: "string".to_owned(), + }, + ); + + // Add source information + attributes.insert( + "log_source".to_owned(), + SentryLogAttribute { + value: serde_json::Value::String("sentry-cli-tail".to_owned()), + attr_type: "string".to_owned(), + }, + ); + attributes.insert( + "organization".to_owned(), + SentryLogAttribute { + value: serde_json::Value::String(self.org.clone()), + attr_type: "string".to_owned(), + }, + ); + attributes.insert( + "project".to_owned(), + SentryLogAttribute { + value: serde_json::Value::String(self.project.clone()), + attr_type: "string".to_owned(), + }, + ); + + // Get timestamp + let timestamp = if let Some(ts) = log_entry.timestamp { + ts.timestamp() as f64 + (ts.timestamp_subsec_nanos() as f64 / 1_000_000_000.0) + } else { + SystemTime::now() + .duration_since(UNIX_EPOCH) + .unwrap_or_default() + .as_secs_f64() + }; + + // Generate a random trace_id (32 hex chars) + let trace_id = format!("{:032x}", rand::random::()); + + let (level, severity_number) = convert_log_level_to_sentry(&log_entry.level); + + SentryLogPayload { + timestamp, + trace_id, + level, + body: log_entry.message.clone(), + attributes, + severity_number: Some(severity_number), + } + } + + /// Create a plain text Sentry log entry + fn create_plain_text_sentry_log(&self, message: &str) -> SentryLogPayload { + let mut attributes = HashMap::new(); + + // Add SDK information + attributes.insert( + "sentry.sdk.name".to_owned(), + SentryLogAttribute { + value: serde_json::Value::String("sentry-cli".to_owned()), + attr_type: "string".to_owned(), + }, + ); + attributes.insert( + "sentry.sdk.version".to_owned(), + SentryLogAttribute { + value: serde_json::Value::String(env!("CARGO_PKG_VERSION").to_owned()), + attr_type: "string".to_owned(), + }, + ); + + // Add format information + attributes.insert( + "log_format".to_owned(), + SentryLogAttribute { + value: serde_json::Value::String("plain".to_owned()), + attr_type: "string".to_owned(), + }, + ); + + // Add source information + attributes.insert( + "log_source".to_owned(), + SentryLogAttribute { + value: serde_json::Value::String("sentry-cli-tail".to_owned()), + attr_type: "string".to_owned(), + }, + ); + attributes.insert( + "organization".to_owned(), + SentryLogAttribute { + value: serde_json::Value::String(self.org.clone()), + attr_type: "string".to_owned(), + }, + ); + attributes.insert( + "project".to_owned(), + SentryLogAttribute { + value: serde_json::Value::String(self.project.clone()), + attr_type: "string".to_owned(), + }, + ); + + let timestamp = SystemTime::now() + .duration_since(UNIX_EPOCH) + .unwrap_or_default() + .as_secs_f64(); + + // Generate a random trace_id (32 hex chars) + let trace_id = format!("{:032x}", rand::random::()); + + SentryLogPayload { + timestamp, + trace_id, + level: "info".to_owned(), + body: message.to_owned(), + attributes, + severity_number: Some(9), // Info level is 9-12, we use 9 + } + } + + /// Send a raw envelope to Sentry using the updated envelope API + fn send_raw_envelope(&self, envelope_bytes: Vec) -> Result<()> { + self.envelope_api + .send_raw_envelope(envelope_bytes) + .context("Failed to send raw log envelope to Sentry")?; + Ok(()) + } +} + +/// Convert our LogLevel to Sentry log level and severity number +/// Returns (level_string, severity_number) as per Sentry logs protocol +fn convert_log_level_to_sentry(level: &Option) -> (String, u8) { + match level { + Some(LogLevel::Debug) => ("debug".to_owned(), 5), // Debug level: 5-8 + Some(LogLevel::Info) => ("info".to_owned(), 9), // Info level: 9-12 + Some(LogLevel::Warning) => ("warn".to_owned(), 13), // Warn level: 13-16 + Some(LogLevel::Error) => ("error".to_owned(), 17), // Error level: 17-20 + Some(LogLevel::Fatal) => ("fatal".to_owned(), 21), // Fatal level: 21-24 + None => ("info".to_owned(), 9), // Default to info for unspecified levels + } +} + +/// Rate limiter for controlling log transmission frequency +pub struct TransmissionRateLimiter { + max_events_per_minute: u32, + current_minute: DateTime, + events_this_minute: u32, +} + +impl TransmissionRateLimiter { + /// Create a new rate limiter + pub fn new(max_events_per_minute: u32) -> Self { + TransmissionRateLimiter { + max_events_per_minute, + current_minute: Utc::now(), + events_this_minute: 0, + } + } + + /// Check if we can send more events, and update counters + pub fn can_send(&mut self) -> bool { + let now = Utc::now(); + + // Reset counter if we've moved to a new minute + use chrono::Timelike as _; + if now.minute() != self.current_minute.minute() || now.hour() != self.current_minute.hour() + { + self.current_minute = now; + self.events_this_minute = 0; + } + + if self.events_this_minute >= self.max_events_per_minute { + false + } else { + self.events_this_minute += 1; + true + } + } + + /// Get current rate limit status + pub fn get_status(&self) -> (u32, u32) { + (self.events_this_minute, self.max_events_per_minute) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::utils::log_parsing::LogLevel; + + #[test] + fn test_convert_log_level_to_sentry() { + assert_eq!( + convert_log_level_to_sentry(&Some(LogLevel::Debug)), + ("debug".to_owned(), 5) + ); + assert_eq!( + convert_log_level_to_sentry(&Some(LogLevel::Info)), + ("info".to_owned(), 9) + ); + assert_eq!( + convert_log_level_to_sentry(&Some(LogLevel::Warning)), + ("warn".to_owned(), 13) + ); + assert_eq!( + convert_log_level_to_sentry(&Some(LogLevel::Error)), + ("error".to_owned(), 17) + ); + assert_eq!( + convert_log_level_to_sentry(&Some(LogLevel::Fatal)), + ("fatal".to_owned(), 21) + ); + assert_eq!(convert_log_level_to_sentry(&None), ("info".to_owned(), 9)); + } + + #[test] + fn test_rate_limiter() { + let mut limiter = TransmissionRateLimiter::new(2); + + assert!(limiter.can_send()); // 1st event + assert!(limiter.can_send()); // 2nd event + assert!(!limiter.can_send()); // Should be rate limited + + let (current, max) = limiter.get_status(); + assert_eq!(current, 2); + assert_eq!(max, 2); + } +} diff --git a/src/utils/memory_monitor.rs b/src/utils/memory_monitor.rs new file mode 100644 index 0000000000..aa8106f0b2 --- /dev/null +++ b/src/utils/memory_monitor.rs @@ -0,0 +1,241 @@ +#![allow(clippy::allow_attributes)] + +use log::{debug, warn}; +use std::sync::atomic::{AtomicUsize, Ordering}; +use std::sync::Arc; +use std::time::{Duration, Instant}; + +/// Memory usage monitor for tracking and limiting memory consumption +#[derive(Debug, Clone)] +pub struct MemoryMonitor { + /// Current estimated memory usage in bytes + current_usage: Arc, + /// Maximum allowed memory usage in bytes + max_usage: usize, + /// Last time we logged memory stats + last_log_time: Arc, + /// Log interval in seconds + log_interval: Duration, +} + +impl MemoryMonitor { + /// Create a new memory monitor with the specified maximum usage + pub fn new(max_usage_mb: usize) -> Self { + MemoryMonitor { + current_usage: Arc::new(AtomicUsize::new(0)), + max_usage: max_usage_mb * 1024 * 1024, // Convert MB to bytes + last_log_time: Arc::new(AtomicUsize::new(0)), + log_interval: Duration::from_secs(30), + } + } + + /// Record memory usage for a log entry + pub fn record_log_entry(&self, entry_size: usize) -> bool { + loop { + let current_usage = self.current_usage.load(Ordering::Relaxed); + let new_usage = current_usage + entry_size; + + // Check if we're approaching the memory limit before recording + if new_usage > self.max_usage { + warn!( + "Memory usage would exceed limit: {} MB / {} MB", + new_usage / (1024 * 1024), + self.max_usage / (1024 * 1024) + ); + return false; + } + + // Try to atomically update the usage if within limits + match self.current_usage.compare_exchange_weak( + current_usage, + new_usage, + Ordering::Relaxed, + Ordering::Relaxed, + ) { + Ok(_) => { + // Log memory stats periodically + self.maybe_log_stats(new_usage); + return true; + } + Err(_) => { + // Another thread modified the value, retry + continue; + } + } + } + } + + /// Record memory release when entries are flushed + pub fn record_flush(&self, released_bytes: usize) { + let previous = self + .current_usage + .fetch_sub(released_bytes, Ordering::Relaxed); + debug!( + "Released {} bytes, current usage: {} MB", + released_bytes, + (previous - released_bytes) / (1024 * 1024) + ); + } + + /// Get current memory usage in bytes + #[allow(dead_code)] + pub fn current_usage_bytes(&self) -> usize { + self.current_usage.load(Ordering::Relaxed) + } + + /// Get maximum allowed memory usage in MB + #[allow(dead_code)] + pub fn max_usage_mb(&self) -> usize { + self.max_usage / (1024 * 1024) + } + + /// Get memory usage percentage (0-100) + pub fn usage_percentage(&self) -> f64 { + let current = self.current_usage.load(Ordering::Relaxed) as f64; + let max = self.max_usage as f64; + (current / max) * 100.0 + } + + /// Maybe log memory statistics if enough time has passed + fn maybe_log_stats(&self, current_usage: usize) { + let now = Instant::now().elapsed().as_secs() as usize; + let last_log = self.last_log_time.load(Ordering::Relaxed); + + if now - last_log >= self.log_interval.as_secs() as usize + && self + .last_log_time + .compare_exchange(last_log, now, Ordering::Relaxed, Ordering::Relaxed) + .is_ok() + { + debug!( + "Memory usage: {} MB / {} MB ({:.1}%)", + current_usage / (1024 * 1024), + self.max_usage / (1024 * 1024), + self.usage_percentage() + ); + } + } +} + +/// Estimate the memory footprint of a string entry +pub fn estimate_entry_size(entry: &str) -> usize { + // Base string size + some overhead for Vec storage and metadata + entry.len() + std::mem::size_of::() + 32 +} + +/// Memory-bounded queue for log entries with automatic cleanup +#[derive(Debug)] +#[allow(dead_code)] +pub struct BoundedLogQueue { + entries: Vec, + memory_monitor: MemoryMonitor, + max_entries: usize, +} + +impl BoundedLogQueue { + /// Create a new bounded log queue + #[allow(dead_code)] + pub fn new(max_memory_mb: usize, max_entries: usize) -> Self { + BoundedLogQueue { + entries: Vec::with_capacity(max_entries.min(1000)), + memory_monitor: MemoryMonitor::new(max_memory_mb), + max_entries, + } + } + + /// Add an entry to the queue, potentially dropping old entries + #[allow(dead_code)] + pub fn push(&mut self, entry: String) -> bool { + let entry_size = estimate_entry_size(&entry); + + // Check memory limit + if !self.memory_monitor.record_log_entry(entry_size) { + // Memory limit exceeded, drop oldest entries + self.make_room(entry_size); + } + + // Check entry count limit + if self.entries.len() >= self.max_entries { + let removed = self.entries.remove(0); + let removed_size = estimate_entry_size(&removed); + self.memory_monitor.record_flush(removed_size); + } + + self.entries.push(entry); + true + } + + /// Make room by removing old entries + #[allow(dead_code)] + fn make_room(&mut self, needed_bytes: usize) { + let mut freed_bytes = 0; + let target_bytes = needed_bytes + (self.memory_monitor.max_usage_mb() * 1024 * 1024) / 10; // Free 10% extra + + while freed_bytes < target_bytes && !self.entries.is_empty() { + let removed = self.entries.remove(0); + let removed_size = estimate_entry_size(&removed); + freed_bytes += removed_size; + self.memory_monitor.record_flush(removed_size); + } + + warn!( + "Memory pressure: dropped {} entries to free {} bytes", + freed_bytes / estimate_entry_size("average_entry"), + freed_bytes + ); + } + + /// Drain all entries and update memory tracking + #[allow(dead_code)] + pub fn drain(&mut self) -> Vec { + let entries = std::mem::take(&mut self.entries); + let total_size: usize = entries.iter().map(|e| estimate_entry_size(e)).sum(); + self.memory_monitor.record_flush(total_size); + entries + } + + /// Get current queue length + #[allow(dead_code)] + pub fn len(&self) -> usize { + self.entries.len() + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_memory_monitor() { + let monitor = MemoryMonitor::new(1); // 1 MB limit + + // Should accept small entries + assert!(monitor.record_log_entry(1000)); + assert_eq!(monitor.current_usage_bytes(), 1000); + + // Should reject when limit exceeded + assert!(!monitor.record_log_entry(2 * 1024 * 1024)); // 2 MB + + // Should handle flush correctly + monitor.record_flush(500); + assert_eq!(monitor.current_usage_bytes(), 500); + } + + #[test] + fn test_bounded_queue() { + let mut queue = BoundedLogQueue::new(1, 3); // 1 MB, 3 entries max + + queue.push("entry1".to_owned()); + queue.push("entry2".to_owned()); + queue.push("entry3".to_owned()); + assert_eq!(queue.len(), 3); + + // Should drop oldest when adding 4th entry + queue.push("entry4".to_owned()); + assert_eq!(queue.len(), 3); + + let entries = queue.drain(); + assert_eq!(entries.len(), 3); + assert_eq!(entries[0], "entry2"); // entry1 was dropped + } +} diff --git a/src/utils/mod.rs b/src/utils/mod.rs index 226aeac9fb..0d8bee1df9 100644 --- a/src/utils/mod.rs +++ b/src/utils/mod.rs @@ -3,6 +3,7 @@ pub mod android; pub mod appcenter; pub mod args; pub mod auth_token; +pub mod batching; pub mod chunks; pub mod cordova; pub mod dif; @@ -10,16 +11,21 @@ pub mod dif_upload; pub mod event; pub mod file_search; pub mod file_upload; +pub mod file_watcher; pub mod formatting; pub mod fs; pub mod http; +pub mod log_parsing; +pub mod log_transmission; pub mod logging; +pub mod memory_monitor; pub mod metrics; pub mod mobile_app; pub mod progress; pub mod proguard; pub mod releases; pub mod retry; +pub mod sampling; pub mod sourcemaps; pub mod system; pub mod ui; diff --git a/src/utils/sampling.rs b/src/utils/sampling.rs new file mode 100644 index 0000000000..47285a5562 --- /dev/null +++ b/src/utils/sampling.rs @@ -0,0 +1,264 @@ +#![allow(clippy::allow_attributes)] + +use log::debug; +use std::collections::HashMap; +use std::time::{Duration, Instant}; + +/// Adaptive sampling strategy for high-volume log scenarios +#[derive(Debug, Clone)] +pub struct AdaptiveSampler { + /// Current sampling rate (0.0 to 1.0) + current_rate: f64, + /// Base sampling rate when load is normal + base_rate: f64, + /// Minimum sampling rate (never go below this) + min_rate: f64, + /// Maximum sampling rate (never exceed this) + max_rate: f64, + /// Target events per second + target_eps: f64, + /// Recent event count tracking + recent_events: Vec<(Instant, usize)>, + /// Window for tracking recent events + tracking_window: Duration, + /// Counter for decisions since last rate adjustment + decisions_since_adjustment: usize, + /// How often to recalculate the sampling rate + adjustment_interval: usize, +} + +impl AdaptiveSampler { + /// Create a new adaptive sampler + pub fn new(base_rate: f64, target_eps: f64) -> Self { + AdaptiveSampler { + current_rate: base_rate, + base_rate, + min_rate: 0.01, // Always sample at least 1% + max_rate: 1.0, // Never exceed 100% + target_eps, + recent_events: Vec::new(), + tracking_window: Duration::from_secs(60), + decisions_since_adjustment: 0, + adjustment_interval: 100, + } + } + + /// Decide whether to sample this event + pub fn should_sample(&mut self) -> bool { + self.decisions_since_adjustment += 1; + + // Periodically adjust the sampling rate + if self.decisions_since_adjustment >= self.adjustment_interval { + self.adjust_sampling_rate(); + self.decisions_since_adjustment = 0; + } + + // Make sampling decision + let random_value: f64 = rand::random(); + let should_sample = random_value < self.current_rate; + + if should_sample { + self.record_sampled_event(); + } + + should_sample + } + + /// Record that an event was sampled + fn record_sampled_event(&mut self) { + let now = Instant::now(); + self.recent_events.push((now, 1)); + + // Clean old events outside the tracking window + let cutoff = now - self.tracking_window; + self.recent_events + .retain(|(timestamp, _)| *timestamp > cutoff); + } + + /// Adjust sampling rate based on recent activity + fn adjust_sampling_rate(&mut self) { + let current_eps = self.calculate_current_eps(); + let previous_rate = self.current_rate; + + if current_eps > self.target_eps * 1.2 { + // Too many events, decrease sampling rate + self.current_rate = (self.current_rate * 0.8).max(self.min_rate); + debug!( + "High load detected ({:.1} eps), reducing sampling rate: {:.3} -> {:.3}", + current_eps, previous_rate, self.current_rate + ); + } else if current_eps < self.target_eps * 0.5 && self.current_rate < self.base_rate { + // Low load, can increase sampling rate back towards base + self.current_rate = (self.current_rate * 1.2) + .min(self.base_rate) + .min(self.max_rate); + debug!( + "Low load detected ({:.1} eps), increasing sampling rate: {:.3} -> {:.3}", + current_eps, previous_rate, self.current_rate + ); + } + } + + /// Calculate current events per second + fn calculate_current_eps(&self) -> f64 { + if self.recent_events.is_empty() { + return 0.0; + } + + let now = Instant::now(); + let events_in_window: usize = self + .recent_events + .iter() + .filter(|(timestamp, _)| now.duration_since(*timestamp) <= self.tracking_window) + .map(|(_, count)| count) + .sum(); + + events_in_window as f64 / self.tracking_window.as_secs_f64() + } + + /// Get current sampling rate + pub fn current_rate(&self) -> f64 { + self.current_rate + } + + /// Get sampling statistics + #[allow(dead_code)] + pub fn get_stats(&self) -> SamplingStats { + SamplingStats { + current_rate: self.current_rate, + } + } +} + +/// Statistics about sampling behavior +#[derive(Debug, Clone)] +#[allow(dead_code)] +pub struct SamplingStats { + pub current_rate: f64, +} + +/// Priority-based sampling for different log levels +#[derive(Debug)] +pub struct PrioritySampler { + /// Sampling rates by log level priority + level_rates: HashMap, + /// Default sampling rate for unknown levels + default_rate: f64, + /// Adaptive sampler for overall rate control + adaptive_sampler: AdaptiveSampler, +} + +impl PrioritySampler { + /// Create a new priority sampler + pub fn new(target_eps: f64) -> Self { + let mut level_rates = HashMap::new(); + level_rates.insert("fatal".to_owned(), 1.0); // Always sample fatal + level_rates.insert("error".to_owned(), 0.8); // Sample most errors + level_rates.insert("warning".to_owned(), 0.4); // Sample some warnings + level_rates.insert("info".to_owned(), 0.1); // Sample few info + level_rates.insert("debug".to_owned(), 0.01); // Sample very few debug + + PrioritySampler { + level_rates, + default_rate: 0.1, + adaptive_sampler: AdaptiveSampler::new(0.2, target_eps), + } + } + + /// Decide whether to sample based on log level and adaptive sampling + pub fn should_sample(&mut self, log_level: Option<&str>) -> bool { + // Get base rate for this log level + let level_rate = log_level + .and_then(|level| self.level_rates.get(level)) + .copied() + .unwrap_or(self.default_rate); + + // Apply adaptive sampling on top of level-based sampling + let adaptive_factor = self.adaptive_sampler.current_rate(); + let final_rate = level_rate * adaptive_factor; + + // Make sampling decision + let should_sample = rand::random::() < final_rate; + + if should_sample { + // Update adaptive sampler + self.adaptive_sampler.should_sample(); + } + + should_sample + } +} + +/// Simple deterministic sampling based on hash +#[allow(dead_code)] +pub fn hash_sample(content: &str, rate: f64) -> bool { + if rate >= 1.0 { + return true; + } + if rate <= 0.0 { + return false; + } + + use std::collections::hash_map::DefaultHasher; + use std::hash::{Hash as _, Hasher as _}; + + let mut hasher = DefaultHasher::new(); + content.hash(&mut hasher); + let hash = hasher.finish(); + + let threshold = (rate * (u64::MAX as f64)) as u64; + hash < threshold +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_adaptive_sampler() { + let mut sampler = AdaptiveSampler::new(0.5, 10.0); + + // Initial rate should be the base rate + assert!((sampler.current_rate() - 0.5).abs() < 0.01); + + // Should make sampling decisions + let _ = sampler.should_sample(); + let stats = sampler.get_stats(); + assert!(stats.current_rate > 0.0); + } + + #[test] + fn test_priority_sampler() { + let mut sampler = PrioritySampler::new(10.0); + + // Fatal errors should have higher sampling rate + let fatal_decisions: Vec<_> = (0..100) + .map(|_| sampler.should_sample(Some("fatal"))) + .collect(); + let fatal_rate = fatal_decisions.iter().filter(|&&x| x).count() as f64 / 100.0; + + let debug_decisions: Vec<_> = (0..100) + .map(|_| sampler.should_sample(Some("debug"))) + .collect(); + let debug_rate = debug_decisions.iter().filter(|&&x| x).count() as f64 / 100.0; + + // Fatal should be sampled more often than debug + assert!(fatal_rate > debug_rate); + } + + #[test] + fn test_hash_sample() { + // Test with 50% rate + let samples: Vec<_> = (0..1000) + .map(|i| hash_sample(&format!("entry_{i}"), 0.5)) + .collect(); + + let sample_rate = samples.iter().filter(|&&x| x).count() as f64 / 1000.0; + + // Should be roughly 50% (within 10% tolerance) + assert!((sample_rate - 0.5).abs() < 0.1); + + // Same content should always give same result + assert_eq!(hash_sample("test", 0.5), hash_sample("test", 0.5)); + } +} diff --git a/tests/integration/_cases/logs/logs-help.trycmd b/tests/integration/_cases/logs/logs-help.trycmd index a356a4c742..72f8b947ed 100644 --- a/tests/integration/_cases/logs/logs-help.trycmd +++ b/tests/integration/_cases/logs/logs-help.trycmd @@ -10,6 +10,7 @@ Usage: sentry-cli[EXE] logs [OPTIONS] [COMMAND] Commands: list [BETA] List logs from your organization + tail [BETA] Monitor log files in real-time and send entries to Sentry help Print this message or the help of the given subcommand(s) Options: