Log Analysis System

A microservices-based log analysis and observability platform that simulates a distributed food-delivery system, ingests structured logs in real time, detects anomalies, and visualizes insights through a React dashboard and the ELK stack.

---

Overview

This project demonstrates an end-to-end observability pipeline for a multi-service application. It generates synthetic logs from six Flask microservices, ships them to Elasticsearch via Logstash, applies statistical and ML-based anomaly detection, and surfaces the results through a React frontend and Kibana.

The system is designed to showcase practical skills in distributed systems, observability, log processing, anomaly detection, and full-stack dashboard development.

---

Architecture

┌────────────────────────────────────────────────────────────┐
│                     Microservices Layer                    │
│  user_service │ restaurant_service │ order_service         │
│  payment_service │ delivery_service │ notification_service │
│  (Flask + structured JSON logging + Prometheus metrics)    │
└──────────────────────────┬──────────────────────────────── ┘
                           │ logs
                           ▼
┌───────────────────────────────────────────────────────────┐
│                      Log Ingestion Layer                  │
│  Logstash  →  Grok + JSON parsing  →  Elasticsearch       │
└──────────────────────────┬────────────────────────────────┘
                           │ query / analyze
                           ▼
┌─────────────────────────────────────────────────────────────┐
│                     Analytics & ML Layer                    │
│  Anomaly detection  │  Request journey analysis  │  Prophet │
│  Rule engine        │  Dynamic thresholds        │  Forecast│
└──────────────────────────┬──────────────────────────────────┘
                           │ visualize
                           ▼
┌─────────────────────────────────────────────────────────────┐
│                     Visualization Layer                     │
│  React + Vite + Tailwind frontend dashboard                 │
│  Kibana for Elasticsearch-native exploration                │
└─────────────────────────────────────────────────────────────┘

---

Features

• Six Flask microservices simulating user, restaurant, order, payment, delivery, and notification domains.
• Structured JSON logging enriched with request metadata: response time, HTTP status, CPU/memory usage, request/trace/span IDs, and environment tags.
• Logstash ingestion pipeline that parses logs via Grok, normalizes JSON, and indexes them into Elasticsearch with daily rotated indices.
• ELK stack (Elasticsearch + Kibana + Logstash) for storage, search, and visualization.
• Synthetic dataset generator (dataset..py) producing 10,000+ realistic log records for testing and model development.
• ML anomaly detection:
  • Response time spike detection using dynamic hybrid thresholds (clustering + EVT).
  • Response time pattern change detection via linear regression and sliding-window slope analysis.
  • Error rate anomaly detection at the 99th percentile.
  • End-to-end request journey risk scoring.
  • Forecasting with Facebook Prophet for response time and error rate trends.
• Rule engine with composable rules, dynamic thresholds, and compound alerting.
• React dashboard (frontend/api-firepower-panel) for interactive visualization of logs and anomalies.
• ServiceNow/Slack integration hooks for incident notification workflows.
• Prometheus metrics exposed on each microservice (/metrics endpoint).

---

Tech Stack

Layer	Technologies
Microservices	Python, Flask, Prometheus Client
Log Ingestion	Logstash, Elasticsearch
Visualization	Kibana, React, Vite, Tailwind CSS
Data Science	pandas, NumPy, scikit-learn, Prophet, scipy, matplotlib
Deployment	Docker, Docker Compose
Frontend Tooling	ESLint, PostCSS, Axios

---

Project Structure

Log-Analysis-System/
├── docker-compose.yml          # Full local stack orchestration
├── logstash.conf               # Logstash pipeline: grok → JSON parse → Elasticsearch
├── parsers.conf                # Fluent Bit style parser configuration
├── dataset..py                 # Synthetic log dataset generator
│
├── user_service/               # Flask microservices
├── restaurant_service/
├── order_service/
├── payment_service/
├── delivery_service/
└── notification_service/
│
├── Automated/                  # Production-style anomaly detection pipeline
│   ├── automodel.py            # Dynamic threshold models + rule engine
│   ├── pipeline.py             # Elasticsearch polling + anomaly push loop
│   ├── RuleEngine.py           # Rule evaluation engine
│   ├── ModelDashboard.py       # Dashboard for model results
│   └── incident_manager.py     # Incident triage logic
│
├── finalML/                    # ML experimentation and forecasting
│   ├── model.py                # Prophet forecasting + anomaly detection
│   ├── data.py                 # Data utilities
│   └── vi.py                   # Visualization helpers
│
├── FINAL FINAL/               # Final deliverables / task submissions
│
└── frontend/api-firepower-panel/  # React + Vite dashboard
    ├── package.json
    ├── src/
    └── README.md

---

Quick Start

Prerequisites

• Docker & Docker Compose
• Python 3.10+ (for standalone ML scripts)
• Node.js 18+ (for frontend development)

Run the full stack

# Clean previous volumes and rebuild
docker-compose down -v
docker-compose build --no-cache
docker-compose up

Once running, the following services are available:

Service	URL
Elasticsearch	http://localhost:9200
Kibana	http://localhost:5601
Logstash Beats Input	http://localhost:5044
User Service	http://localhost:5001
Restaurant Service	http://localhost:5002
Order Service	http://localhost:5003
Payment Service	http://localhost:5004
Delivery Service	http://localhost:5005
Notification Service	http://localhost:5006

Run the frontend dashboard

cd frontend/api-firepower-panel
npm install
npm run dev

Generate synthetic data

python dataset..py
# Output: synthetic_full_datasetlakh.csv

Run the anomaly detection pipeline

cd Automated
python pipeline.py

---

ML & Anomaly Detection

The analytics layer covers three core anomaly types:

1. Response Time Spikes — detects when an endpoint's average response time exceeds a dynamic threshold computed via clustering (K-Means) and Extreme Value Theory (GPD).
2. Pattern Changes — identifies sustained upward trends in response time using linear regression on sliding windows.
3. Error Rate Anomalies — flags error rate bursts using hybrid percentile thresholds over 2-hour windows.

Additionally, the system performs request journey analysis by grouping requests across services, computing a composite risk score, and forecasting anomalous journey counts with Prophet.

---

Authors & Contributors

• Prathu Vijayvargiya (Prathuvj)
• Shivam Bansal (ShivamBansal07)
• pixmania2

---

License

This project is for educational and demonstration purposes. See repository for licensing details.