EAI-7555_Overhaul_lgtm_stack

[woojae-siloai]

Overview

Upgrades LGTM stack from v1.0.7 to v1.0.8, migrating to the official Grafana image and introducing GitOps-managed configuration overrides.

Key Changes

1. Official Grafana Image Migration

• Before: ghcr.io/silogen/docker-otel-lgtm:v1.0.7 (our own custom)
• After: grafana/otel-lgtm:0.29.2 (official)

Benefits: Eliminates custom image maintenance, automatic upstream patches, latest components (Grafana 13.1, Loki 3.7.3, Prometheus 3.13.1), simplified upgrades.

2. Configurable Loki & OtelCol

ConfigMap-based overrides for production-ready settings:

Loki:

• 7-day retention with compaction prevents unbounded growth
• Balanced I/O: 30m compaction, 10 workers

3. Kubernetes Improvements

• Faster deploys: Probe delays 180s → 60s/120s (pod ready in ~60-75s)
• Auto rollouts: ConfigMap checksum triggers automatic pod restarts
• Fixed health checks: 6 actual services vs. 11 wrapper scripts

Benefits

• Cost control: 7-day log retention vs. unlimited (automatically enforced)
• Upgrades: Change image tag, no rebuild

Validated on a VM—all services operational, configs verified.