DevOps MultiCloud Project

This repository implements a multi-cloud DevOps architecture with an AWS primary deployment and an Azure disaster recovery (DR) deployment.

Architecture Overview

High-Level Diagram

Overview

• AWS primary: Terraform deploys VPC, EKS, EC2, ECR, EFS, and supporting networking/security.
• Azure DR: Terraform deploys Resource Group, VNet, AKS, ACR, Azure Files, private endpoints, and Traffic Manager.
• Traffic Manager: Routes traffic to AWS (priority 1) or Azure (priority 2) based on health checks
• Ansible: Configures the Jenkins VM, installs Docker, Jenkins, kubectl/Helm/Azure CLI, and sets up ArgoCD.
• CI/CD: Jenkins pipelines build Docker images, scan them with Trivy, push to registry, and deploy to Kubernetes.
• GitOps: ArgoCD deploys app overlays for AWS and Azure.
• Monitoring: Prometheus and Grafana

Repository Structure

• terraform/aws/

  • main.tf root AWS deployment
  • modules/networking/ VPC, subnets, NAT, route tables, endpoints
  • modules/compute/ EKS, node group, EC2 Jenkins, EFS, IAM, ECR
  • outputs.tf, variables.tf, envs/ variables files

• terraform/azure/

  • main.tf root Azure deployment
  • modules/networking/ VNet, subnets, private endpoints
  • modules/compute/ AKS, VM, storage, Azure Files, private endpoint
  • modules/security/ NSGs, ACR, private DNS, security rules
  • outputs.tf, variables.tf, envs/ variables files

• ansible/

  • inventory/ AWS/Azure inventory files
  • playbooks/ Ansible playbooks for AWS and Azure
  • roles/ reusable configuration roles for Jenkins, Kubernetes, ArgoCD, Docker, etc.

• jenkins/

  • Jenkinsfile-aws CI/CD pipeline for AWS EKS
  • Jenkinsfile-azure CI/CD pipeline for Azure AKS

• kubernetes/

  • argocd/ ArgoCD application manifests for AWS and Azure
  • overlays/aws/ and overlays/azure/ Kustomize overlays

Getting Started

AWS

1. Change to the AWS Terraform directory:

   cd terraform/aws

2. Initialize Terraform:

   terraform init

3. Plan or apply with the environment variables file:

   terraform plan -var-file=envs/prod.tfvars
   terraform apply -var-file=envs/prod.tfvars

4. Use Ansible to configure the EC2/Jenkins host:

   ANSIBLE_CONFIG=aws.cfg ansible-playbook -i ansible/inventory/aws.ini ansible/playbooks/setup-ec2.yml -e "eks_cluster_name=eks-prod-dr eks_region=eu-north-1" --ask-vault-pass -v

Azure

1. Change to the Azure Terraform directory:

   cd terraform/azure

2. Initialize Terraform:

   terraform init

3. Plan or apply using the Azure vars file:

   terraform plan -var-file=envs/prod.tfvars
   terraform apply -var-file=envs/prod.tfvars

4. Use Ansible to configure the Azure Jenkins/AKS DR host:

   ANSIBLE_CONFIG=azure.cfg ansible-playbook -i ansible/inventory/azure.ini ansible/playbooks/setup-azure.yml --ask-vault-pass -v

Jenkins Pipelines

AWS Pipeline Flow

• jenkins/Jenkinsfile-aws
  • builds and scans Docker image
  • pushes to AWS ECR
  • updates kubernetes/overlays/aws/kustomization.yaml
  • deploys to EKS

Azure Pipeline Flow

• jenkins/Jenkinsfile-azure
  • builds and scans Docker image
  • pushes to Azure ACR
  • updates kubernetes/overlays/azure/kustomization.yaml
  • deploys to AKS

GitOps / ArgoCD

• AWS app manifest: kubernetes/argocd/portfolio-aws.yaml
• Azure app manifest: kubernetes/argocd/portfolio-azure.yaml
• ArgoCD deploys the corresponding overlay for the target cluster.

Data Persistence

AWS: Jenkins Data on EFS (Elastic File System)

The Jenkins VM stores all its data on Amazon EFS, which is separate from the EC2 instance's root volume. This ensures that if the EC2 instance fails, crashes, or is terminated, all Jenkins data is preserved.

What EFS stores:

• All Jenkins jobs and build history
• Jenkins plugins and configurations
• Build artifacts and logs
• Credentials and secrets (encrypted)

How it's implemented:

• EFS filesystem created in terraform/aws/modules/preserved_efs/main.tf (separate module to preserve lifecycle)
• EFS security group allows NFS traffic (port 2049) only from the Jenkins instance
• Ansible mounts EFS to /var/lib/jenkins before starting Jenkins (ansible/roles/jenkins/tasks/main.yml)
• Jenkins service reads/writes all data directly to the mounted EFS

Azure: Jenkins Data on Azure Files (SMB Share)

Azure's disaster recovery deployment uses Azure Files to provide the same data persistence guarantee as AWS EFS.

What Azure Files stores:

• Same as AWS: all Jenkins jobs, configurations, plugins, and build artifacts
• Encrypted SMB 3.x connection between VM and Storage Account

How it's implemented:

• Azure Storage Account created in terraform/azure/modules/compute/main.tf
• Azure Files share (jenkins-share) with 100GB quota for Jenkins home
• Private endpoint attached to Storage Account (secure VNet access)
• Ansible mounts the SMB share to /var/lib/jenkins on the Azure VM
• Identity-based authentication (no storage key hardcoded)

---

Monitoring & Observability with Prometheus & Grafana

Prometheus continuously collects and stores metrics

Grafana transforms Prometheus metrics into actionable dashboards:

Dashboard 1:

Dashboard 2:

Dashboard 3:

Storage Monitoring

EFS Monitoring (AWS)

• Why it matters: Detects if EFS is running out of space before Jenkins fills up
• Alert triggers: If EFS usage > 80% or throughput at limits, alert ops team

GitOps Monitoring

ArgoCD Application State

• Sync status: Application matches Git repo or out of sync
• Health status: All pods running and healthy
• Deployment history: Last sync time and changes

Implementation in the Project

Prometheus Setup:

• Installed via ansible/roles/monitoring/ playbook
• Scrapes metrics from Kubernetes API server, kubelet, node exporter
• Retains 15 days of historical metrics data
• Persistent volume stores metrics (survives pod restarts)

Grafana Setup:

• Installed on the same monitoring namespace
• Pre-configured with Prometheus datasource
• Pre-built dashboards for Kubernetes, Jenkins, and application monitoring
• Alert rules configured for critical thresholds (pod crashes, high latency, etc.)

Metric Retention Policy:

• High-resolution metrics (15-second intervals): 7 days
• Low-resolution metrics (1-hour intervals): 15 days
• Alerts fired when thresholds breached (sent to Slack/email)

---

Security Architecture

How the Project Secures Access

The infrastructure is locked down — Nobody can access it without proper authorization. Here's the multi-layered security implementation:

1. Network Isolation — Private Subnets (No Public Access)

AWS Implementation:

• EKS cluster worker nodes run in private subnets → No direct internet access
• EC2 Jenkins VM runs in private subnet → Cannot be SSH'd from internet
• Internet Gateway only attached to public subnets → No route to instances
• NAT Gateway provides outbound-only access → Instances can reach internet, but internet cannot reach instances

Azure Implementation:

• AKS nodes in private subnet → No public IPs assigned
• Jenkins VM in private subnet → No direct internet connection
• Bastion subnet for administrative access → Only admins can SSH via bastion host
• User-Defined Routes (UDRs) enforce traffic through bastion or private endpoints

Result: The infrastructure is completely invisible to the internet — nobody can scan or discover it.

How it's built in the project:

• terraform/aws/modules/networking/main.tf — defines private subnets with no route to IGW
• terraform/azure/modules/networking/main.tf — defines AKS private subnet + bastion access pattern

---

2. Security Groups & Network Security Groups (Firewall Rules)

AWS Security Groups (stateful firewall):

• EKS control plane SG:

  • Blocks all unsolicited traffic
  • Only allows HTTPS (443) from worker nodes
  • Only allows API calls from within the cluster

• EKS worker node SG:

  • Blocks all inbound from internet
  • Allows node-to-node communication (kubelet, kube-proxy)
  • Allows EKS control plane → node communication
  • Allows pods to reach external APIs

• Jenkins EC2 SG:

  • Blocks SSH from internet
  • Only allows traffic from within VPC
  • Allows Jenkins to reach Docker registry (ECR), Kubernetes API
  • Never exposes port 8080 to internet

• EFS SG:

  • Blocks NFS from everywhere
  • Only allows NFS (port 2049) from Jenkins instance SG

Azure Network Security Groups (stateful firewall):

• Private subnet NSG:

  • Denies all inbound from internet
  • Allows VNet-to-VNet communication
  • Blocks inter-subnet traffic unless explicitly allowed

• AKS subnet NSG:

  • Blocks SSH/RDP from anywhere
  • Only allows Kubernetes control plane traffic
  • Allows pod-to-service communication within cluster

• Bastion NSG:

  • SSH (22) allowed ONLY from admin IPs (configured in variables)
  • All other inbound denied
  • Outbound to private resources allowed

Result: Layered firewall protection — Even if someone breaches the network perimeter, microsegmentation prevents them from reaching your services.

How it's built in the project:

• terraform/aws/modules/networking/main.tf — Security groups with explicit allow rules
• terraform/azure/modules/security/main.tf — NSGs with ingress/egress rules

---

3. Identity & Access Management (IAM) — Least Privilege Principle

AWS IAM Roles (fine-grained permissions):

• role-ec2-prod (Jenkins EC2 instance role):

  • ec2:DescribeInstances — read-only cluster info
  • ecr:GetDownloadUrlForLayer — pull Docker images
  • logs:PutLogEvents — send logs to CloudWatch
  • ssm:StartSession — secure shell via Session Manager (no SSH key needed)
  • CANNOT create/delete resources
  • CANNOT access other AWS accounts
  • CANNOT escalate privileges

• role-eks-nodes-prod (EKS worker node role):

  • ec2:AssignPrivateIpAddresses — for VPC CNI plugin (networking)
  • ecr:GetAuthorizationToken — pull container images from ECR
  • eks:DescribeCluster — read cluster configuration
  • cloudwatch:PutMetricData — send metrics
  • CANNOT launch new instances
  • CANNOT modify security groups
  • CANNOT access S3 or other services

• role-eks-cluster-prod (EKS control plane role):

  • ec2:CreateNetworkInterface — create ENIs for pods
  • elasticloadbalancing:* — manage load balancers for services
  • logs:CreateLogGroup — create CloudWatch log groups
  • CANNOT manage IAM or other privileged operations

Result: Each component has ONLY the minimum permissions needed — If credentials leak, attacker can only do limited damage.

Azure Managed Identity (role-based access):

• AKS system-assigned managed identity:

  • Automatically rotated credentials
  • No secrets stored in code or configuration
  • Azure RBAC controls what the identity can do
  • Cannot create storage accounts or other resources

• AKS kubelet identity (for ACR pull):

  • Dedicated identity for pod authentication
  • ACR pull access (download images only)
  • Cannot push images or modify registry

• Service Principal (for Terraform/Ansible):

  • Stored in Ansible Vault (encrypted)
  • Required at runtime (--ask-vault-pass)
  • Minimal permissions for infrastructure changes
  • Not stored in code or environment variables

Result: No hardcoded credentials — All identities use managed services and encrypted storage.

How it's built in the project:

• terraform/aws/modules/compute/main.tf — IAM role creation with specific policy attachments
• terraform/azure/modules/compute/main.tf — Managed Identity configuration
• ansible/vault.yml — Encrypted secrets (service principal credentials, SSH keys)

---

4. Private Endpoints & VPC Isolation (No Internet Exposure)

AWS VPC Endpoints:

• S3 endpoint: Jenkins/pods reach S3 buckets without going through internet
• ECR endpoint: Pull Docker images through private VPC endpoint (traffic stays in AWS network)
• SSM endpoint: Secure shell to EC2 instances without public IPs
• EKS endpoint: Kubernetes API accessible only within VPC (not public)
• CloudWatch endpoint: Send logs/metrics through private network

Result: No data leaves your VPC — All service calls encrypted and routed internally.

Azure Private Endpoints:

• ACR endpoint: AKS pulls images through private endpoint (no internet required)
• Storage endpoint: Jenkins mounts Azure Files through private endpoint
• Key Vault endpoint: Access secrets through private network (optional)
• AKS endpoint: API server accessible from within VNet only

Result: Container registry completely private — Attackers cannot enumerate or pull images.

How it's built in the project:

• terraform/aws/modules/networking/main.tf — VPC endpoint creation for S3, ECR, SSM
• terraform/azure/modules/security/main.tf — Private endpoint configuration for ACR and Storage

---

5. Encryption in Transit & at Rest

AWS Encryption:

• EFS encryption: Uses AWS KMS (Key Management Service) — data encrypted before writing to disk
• EBS volumes: EC2 root volume encrypted with KMS
• Network encryption:
  • HTTPS/TLS for all Kubernetes API calls
  • TLS for kubelet → API server communication
  • Encrypted SSH tunnels (Session Manager)
• VPC Flow Logs: Capture all network traffic for audit (encrypted logs to CloudWatch)

Azure Encryption:

• Azure Files encryption: SMB 3.x protocol with AES-256-GCM encryption
• Storage Account encryption: Server-side encryption (automatically encrypted at rest)
• VM disk encryption: OS disk encrypted with Azure Disk Encryption
• HTTPS enforcement: Storage account blocks HTTP (HTTPS only)
• Network encryption: All inter-component traffic uses TLS 1.2+

Result: Data protected in flight and at rest — Even if disks are stolen, data is useless without keys.

How it's built in the project:

• terraform/aws/modules/preserved_efs/main.tf — encrypted = true for EFS
• terraform/azure/modules/compute/main.tf — SMB 3.x and encryption settings

---

6. Container Image Security (No Vulnerable Code)

Jenkins Pipelines include Trivy vulnerability scanning:

Jenkins Build Pipeline Flow:
1. Code checked out from Git
2. Build Docker image locally
3. SCAN IMAGE WITH TRIVY ← Detects vulnerabilities before pushing
4. ONLY IF SCAN PASSES → Push to registry (ECR/ACR)
5. ONLY IF PUSH OK → Deploy to Kubernetes

Trivy scanning:

• Detects known CVEs (Common Vulnerabilities and Exposures) in base images and dependencies
• Blocks deployment if critical vulnerabilities found
• Generates SBOM (Software Bill of Materials) for compliance
• Configured in jenkins/Jenkinsfile-aws and jenkins/Jenkinsfile-azure

Registry-level scanning:

• AWS ECR automatically scans images on push
• Azure ACR can be configured for on-push scanning
• Images flagged if vulnerabilities detected post-push

Result: Only security-scanned images reach production — Reduces attack surface.

---

7. Kubernetes RBAC & Network Policies (Pod-to-Pod Isolation)

Kubernetes RBAC (Role-Based Access Control):

• Namespaces: portfolio namespace isolates your application from system components
• Service Accounts: Each deployment uses a dedicated service account (not default)
• Roles: Define what actions a service account can perform (e.g., "read pods" only)
• RoleBindings: Grant roles to service accounts

Result: Even if one pod is compromised, it cannot access other pods or secrets.

Network Policies:

• Egress policies: Pods can only reach specific destinations (e.g., portfolio app → database only)
• Ingress policies: Only specified sources can reach a pod (e.g., load balancer → app pods only)
• Implemented in kubernetes/ manifests with NetworkPolicy resources

Result: Lateral movement blocked — Attacker compromising one pod cannot pivot to others.

How it's built in the project:

• kubernetes/overlays/aws/kustomization.yaml — Kustomize overlays for namespace and deployments
• Network policies defined in deployment manifests

---

8. Jenkins Access Control (Authenticated Builds)

Jenkins Master Security:

• Admin credentials stored in Ansible Vault (encrypted with passphrase)
• Vault password required at runtime: ansible-playbook ... --ask-vault-pass
• Jenkins runs as jenkins user (non-root) — limited system privileges
• GitHub webhook authentication enforced (Jenkins verifies GitHub request signature)

Credential Management:

• Build credentials (Docker registry, AWS keys, etc.) stored in Jenkins Credentials Store
• Encrypted with Jenkins master key (backed up and protected)
• Credentials NEVER appear in logs or build output (masked)
• SSH keys for Git access rotated regularly

Build Pipeline Security:

• Each Jenkinsfile stage can have conditional approvals (require manual click for production deploy)
• Build logs automatically scrubbed of secrets
• Audit log tracks who triggered which build

Result: Only authorized personnel can trigger production deployments.

How it's built in the project:

• ansible/roles/jenkins/tasks/main.yml — Jenkins setup with credential store initialization
• jenkins/Jenkinsfile-* — Pipeline stages with credential references (not hardcoded values)

---

9. Audit Logging (Complete Accountability)

AWS CloudWatch Logs capture:

• VPC Flow Logs: Every network packet source, destination, port, action (ACCEPT/REJECT)
• EKS control plane audit logs: Every API call to Kubernetes (who called what when)
• IAM CloudTrail: Every AWS API call, resource change, user action
• Jenkins build logs: Every build trigger, stage, output (stored 90 days)
• Application logs: Stdout/stderr from all pods (indexed and searchable)

Azure Monitor & Log Analytics capture:

• Flow logs: Network traffic metadata
• Azure Kubernetes Service diagnostic logs: Control plane operations
• Azure Activity Log: Resource group and subscription-level changes
• VM diagnostics: OS-level events and performance

Result: Complete audit trail — If breach occurs, you can trace exactly what happened and when.

Retention Policy:

• Logs retained 1+ years (configurable)
• Searchable and filterable in monitoring dashboards
• Integration with SIEM tools possible (Splunk, DataDog, etc.)

How it's built in the project:

• terraform/aws/modules/compute/main.tf — EKS cluster logging config
• terraform/azure/modules/compute/main.tf — AKS diagnostics settings
• Monitoring role deploys CloudWatch/Log Analytics agents

---

10. Ansible Secrets Management (Secure Automation)

How it's implemented:

• ansible/group_vars/all/vault.yml — Encrypted vault file (unreadable without password)

SSH Key Security:

• Private SSH keys stored locally: ~/.ssh/ with permissions 0400 (owner read-only)
• Public keys added to VM authorized_keys during Terraform apply
• No password-based SSH (key-only authentication)

Result: Secrets encrypted at rest; automation remains safe.

How it's built in the project:

• ansible/playbooks/setup-aws.yml — Loads vault variables
• ansible/roles/jenkins/tasks/main.yml — Uses vault for Jenkins credentials

---

Security Layers Summary Table

Security Layer	AWS Implementation	Azure Implementation	Purpose
Network	Private subnets, NAT Gateway, no IGW	Private subnets, bastion access, NSGs	No internet exposure
Firewall	Security Groups (stateful)	Network Security Groups (stateful)	Microsegmentation
Identity	IAM roles (least privilege)	Managed Identity + RBAC	Fine-grained access control
Privacy	VPC Endpoints (S3, ECR, SSM)	Private Endpoints (ACR, Storage)	Traffic stays in network
Encryption	KMS for EFS/EBS, TLS APIs	SMB 3.x, Azure Disk Encryption	Data protected
Image Security	Trivy scanning in pipeline	Trivy scanning in pipeline	No vulnerable containers
Pod Security	Kubernetes RBAC + network policies	Kubernetes RBAC + network policies	Workload isolation
Access	Jenkins Vault + credentials store	Ansible Vault + Key Vault	Authenticated operations
Audit	CloudWatch + VPC Flow Logs	Azure Monitor + diagnostics	Forensics capability
Secrets	Ansible Vault encryption	Ansible Vault encryption	Credentials never exposed

---