Deep Reinforcement Learning for MiniGrid MultiRoom Navigation

A reinforcement learning project implementing Deep Q-Network (DQN) and Proximal Policy Optimization (PPO) agents to solve the MiniGrid MultiRoom-N6 environment.

Project Overview

This project trains RL agents to navigate through multiple connected rooms in a grid-based environment to reach a goal. The agents must learn to:

• Navigate through 6 interconnected rooms
• Open doors to access new rooms
• Find the optimal path to the goal location

Environment

The MiniGrid MultiRoom-N6 environment features:

• Partially observable grid world (agent sees only a limited view)
• 6 connected rooms with doors
• 4 actions: Turn Left, Turn Right, Move Forward, Toggle (open/close doors)
• RGB image observations processed through CNN

Algorithms Implemented

1. Deep Q-Network (DQN)

• Double DQN: Uses separate networks for action selection and evaluation to reduce overestimation bias
• Dueling Architecture: Separates state value and advantage estimation for better learning
• Prioritized Experience Replay (PER): Samples important transitions more frequently
• Soft target updates: Gradual target network updates using τ parameter

2. Proximal Policy Optimization (PPO)

• Actor-Critic Architecture: Shared CNN backbone for policy and value networks
• Clipped Surrogate Objective: Prevents large policy updates for stable training
• Entropy Bonus: Encourages exploration during training
• Temperature-based Action Selection: Controls exploration/exploitation trade-off

Key Features

CNN Feature Extractor

Both agents use a shared CNN architecture:

• 4 convolutional layers (32 → 64 → 128 → 128 filters)
• ReLU activations
• Global Average Pooling
• Input normalization

Reward Shaping

Custom reward function to accelerate learning:

• +100 for reaching the goal
• +100 for successfully opening a door
• -104 for closing a door
• -3 for bumping into walls or toggling nothing
• -2 for turning
• -1 for moving forward

Results

Agent	Success Rate	Average Reward	95% CI
DQN	79.0%	0.42	[0.37, 0.47]
PPO	81.0%	0.46	[0.41, 0.51]

Evaluated over 100 episodes

Project Structure

├── multiroom_training.ipynb    # Training notebook (run on Google Colab with GPU)
├── multiroom_evaluation.ipynb  # Evaluation and video generation notebook
├── dqn_trained_network.pth     # Trained DQN Q-Network parameters
├── ppo_trained_network.pth     # Trained PPO Actor-Critic parameters
├── explainer.txt               # Instructions for running notebooks
└── README.md                   # This file

Getting Started

Prerequisites

• Python 3.8+
• PyTorch
• Gymnasium
• MiniGrid
• OpenCV (cv2)
• imageio

Running the Evaluation Notebook

1. Download the trained model files:

   • dqn_trained_network.pth
   • ppo_trained_network.pth

2. Open multiroom_evaluation.ipynb in Google Colab

3. Upload the model files to Colab's session storage (Files tab → Upload)

4. Run all cells (Runtime → Run all)

5. Generate agent videos using record_agent_video()

Running the Training Notebook

1. Open multiroom_training.ipynb in Google Colab

2. Enable GPU support (Runtime → Change runtime type → GPU)

3. Run all cells to train both DQN and PPO agents

Hyperparameters

DQN

Parameter	Value
Learning Rate	1e-4
Batch Size	32
Discount Factor (γ)	0.9
Buffer Size	10,000
Update Frequency	10 steps
Soft Update (τ)	0.01
PER Alpha	0.6

PPO

Parameter	Value
Learning Rate	1e-4
Batch Size	32
Discount Factor (γ)	0.9
PPO Clip (ε)	0.2
PPO Steps	8
Entropy Coefficient	0.02

Authors

• Rafael Moreno (ID: 021387121)
• Eran Hadad (ID: 037748431)

References

• MiniGrid Documentation
• DQN Implementation Reference
• PPO Tutorial

License

This project was developed as part of a Deep Reinforcement Learning course mid-semester project.