This repository contains a reinforcement learning solution for retail inventory replenishment. Rather than a simple immediate-feedback environment, this simulator enforces a two-month logistics arrival lag, requiring the agent to make sequential, long-term hedging decisions to avoid the bullwhip effect.
| MDP Component | Definition | Business Logic |
|---|---|---|
| State (S) | s_t = (I_t, P1_t, P2_t) |
A flattened 3D state tracking current inventory (I), stock arriving next month (P1), and stock arriving in two months (P2). Space size: 396 states. |
| Action (A) | a_t ∈ [0, 1, 2, 3, 4, 5] |
The quantity of new inventory ordered today. |
| Reward (R) | R_t = (Rev) - (Cost) - (Hold) - (Pen) |
Net Margin: Revenue ($50) minus unit cost ($20), holding cost ($2), and stockout penalties ($15). |
| Transition (P) | I_{t+1} = max(0, I_t + P1_t - D_t) |
Demand (D) is uniformly stochastic (0-4). Pipeline shifts deterministically: P1_{t+1} = P2_t and P2_{t+1} = a_t. |
| Horizon (T) | T = 24 |
Evaluated over 24-month financial cycles. |
- Baseline (Naive Rule-Based): Orders a fixed batch whenever on-hand inventory drops below 3. It acts myopically, ignoring the 2-month pipeline, leading to massive overstock holding costs.
- Learning Agent (Tabular Q-Learning): An off-policy TD control algorithm. It learns the temporal delay, ordering smaller, smoothed batches to rely on in-transit goods.
To run the simulation and generate the learning curve:
pip install numpy matplotlib
python inventory_agent.py