SIEM Log Analysis Automation (Python)

A modular, lightweight Security Information and Event Management (SIEM) log parser and threat detection engine written in Python. This tool parses authentication logs, web server logs, and general system logs to discover anomalies, audit access, and trigger real-time alerts.

---

Key Features

• 🛠️ Multi-Format Parsers: Implements standardized parsers for:
  • Linux Auth Logs (/var/log/auth.log): Tracks SSH login successes, brute force signatures, invalid users, and sudo privilege escalations.
  • Web Access Logs (Apache/Nginx Combined/Common Formats): Parses URLs, status codes, user-agents, and dynamically URL-decodes incoming paths to prevent obfuscation bypasses.
  • Syslogs (RFC 3164 & RFC 5424): Decodes priority ratings, hostnames, application tags, and records system-level warning/error events.
• 🧠 Stateful Rule Engine:
  • Threshold monitoring (e.g., SSH brute force count limits and HTTP error rate scans over time windows).
  • Regular expression signatures for Web Attacks (SQLi, XSS, Path Traversal, and Command Injection).
  • Sensitive resource tracking (detects queries attempting to access .env, .git, wp-admin, /etc/passwd, etc.).
• 🚀 Two Processing Modes:
  • Batch Analysis: Rapidly scans static log files, aggregates metrics, and builds reports.
  • Real-Time Monitoring (Tailing): Monitors multiple files simultaneously using an open-read-close offset polling strategy that avoids file locking, making it fully compatible with Windows and Linux log-writing daemons.
• 📊 Premium Visual Dashboards & Reporting:
  • Generates detailed, standard machine-readable JSON reports.
  • Generates a self-contained premium HTML dashboard featuring dark-theme glassmorphism, responsive metrics grids, SVG status charts, interactive text searching, and collapsible detailed timeline records.
• 🔔 Multi-Channel Alerting: Writes to a centralized JSON log (alerts.log), outputs alerts to the terminal using ASCII styling, and dispatches rich payload updates to Slack/Discord webhooks.

---

Directory Structure

SIEM_Log_Analysis_Automation/
├── config.yaml               # Settings for rule thresholds, alert webhooks, and signatures
├── requirements.txt          # Python package requirements
├── siem_analyzer.py          # CLI Orchestrator and entry-point script
├── generate_mock_logs.py     # Script to generate sample log datasets for testing
├── parsers/                  # Parsing package
│   ├── __init__.py           
│   ├── base_parser.py        # Abstract base class
│   ├── auth_log_parser.py    # SSH & Sudo log parser
│   ├── web_log_parser.py     # Apache/Nginx web parser (with URL decoding)
│   └── syslog_parser.py      # Syslog RFC 3164/5424 parser
├── rules/                    # Threat detection package
│   ├── __init__.py
│   └── engine.py             # Rule matching, state tracker, and sliding window checks
├── reporters/                # Reporting package
│   ├── __init__.py
│   ├── cli_reporter.py       # Console rich ASCII summary grid
│   ├── json_reporter.py      # JSON writer
│   └── html_reporter.py      # HTML responsive dashboard report generator
└── utils/                    # Helper utilities package
    ├── __init__.py
    ├── notifier.py           # Local logs and Slack/Discord webhook alerts dispatcher
    └── log_tailer.py         # Non-locking multi-file tailing polling loop

---

Installation & Setup

1. Clone or navigate to the project directory:

   cd SIEM_Log_Analysis_Automation

2. Create and activate a virtual environment (recommended):

   python -m venv venv
   # On Windows:
   .\venv\Scripts\activate
   # On Linux/macOS:
   source venv/bin/activate

3. Install dependencies:

   pip install -r requirements.txt

---

Quick Start & Usage

1. Generate Test Datasets

You can generate sample mock log files representing normal operations mixed with attack patterns (brute force, XSS, SQLi, system crashes):

python generate_mock_logs.py

This generates mock_auth.log, mock_web.log, and mock_syslog.log in the local directory.

2. Run Batch Scan

Run a batch scan on the mock files. The tool is configured to automatically discover the mock log files in the current folder if no files are explicitly supplied:

python siem_analyzer.py --output-html security_report.html --output-json security_report.json

Or target specific operational log paths:

python siem_analyzer.py --auth /var/log/auth.log --web /var/log/nginx/access.log --output-html my_report.html

3. Run Real-Time Tailing (Live Monitor)

Monitor log files live as they write. It prints live events to the console, displays high-visibility alert panels when threats occur, and generates reports upon exit (Ctrl+C):

python siem_analyzer.py --auth mock_auth.log --web mock_web.log --tail --output-html live_report.html

---

Configuration (config.yaml)

Edit the config.yaml file to customize security parameters:

• Brute Force: Modify max_failed_attempts and window_seconds to control SSH alert sensitivity.
• Sensitive Access: Add directories/files you wish to restrict (e.g., /admin, /private.key).
• Web Signatures: Tweak regexes under injection_detection for SQLi, XSS, and command payloads.
• Error Rates: Set max_errors and window_seconds to detect automated site scanners (like Gobuster or Dirbuster) getting 404 error spikes.
• Alert Channels: Configure alert_log_file, toggle console display, or enable a Slack/Discord webhook URL to receive live alerts.