Predator-Prey-Grass

Emerging coevolution, cooperation, defection and free-riding through multi-agent deep reinforcement learning

Legacy snapshot: the pre-cleanup research codebase is archived at PredPreyGrassLegacy.

This project explores how cooperative behavior emerges and stabilizes in a spatial, resource-limited ecosystem by combining within-lifetime multi-agent reinforcement learning with population-level ecological and evolutionary dynamics. It explores the interplay between nature (inherited traits via reproduction and mutation) and nurture (behavior learned via reinforcement learning) in ecological systems. We combine Multi-Agent Deep Reinforcement Learning (MADRL) with evolutionary dynamics to explore emergent behaviors in a multi-agent dynamic ecosystem of Predators, Prey, and regenerating Grass. Agents differ by speed, vision, energy metabolism, and decision policies—offering ground for open-ended adaptation. At its core lies a gridworld simulation where agents are not just trained—they are born, age, reproduce, die, and even mutate in a continuously changing environment. This gives rise to coevolution, cooperation, defection and free-riding.

Emerging coevolution, cooperation, defection and free-riding

Environment:

• Stag hunt with defection : Humans can hunt solo for rabbits but mammoths usually cannot be killed alone, so they have decide to cooperate at an energy cost or to defect at zero cost, giving opportunities for free-riding. (implementation)

Darwinian/Baldwinian evolutionary environments

These environments layer a genuine evolutionary algorithm — founder genome, mutation, inheritance — on top of shared-policy PPO. A scalar trait is passed from parent to offspring with mutation at each reproduction event; PPO policy weights are never inherited, only shared per species. Learned behavior (Baldwinian) determines which trait values survive to reproduce, closing a genome → phenotype → learned behavior → fitness → genome-frequency loop across generations — see Darwinian vs. Baldwinian evolution below for the underlying theory.

• Eco-evolutionary: baseline of the family. Evolves a speed trait that sets a movement-distance threshold (1 vs. 2 tiles per move).

• Eco-evolutionary cadence: evolves the same speed trait, expressed as a graded movement cooldown instead of a discrete distance threshold.

• Eco-evolutionary cooperation: evolves a cooperation_rate trait — the fraction of an agent's net energy gain donated to nearby same-species agents, relying on spatial viscosity (offspring spawn near parents) for implicit kin selection.

• Eco-evolutionary investment: evolves an offspring_investment_fraction trait — how much energy a parent hands each offspring at birth.

• Eco-evolutionary metabolic rate: evolves a metabolic_rate trait that symmetrically scales both energy gain and basal energy cost.

• "Stag hunt" nature + nurture: a hybrid case — predators carry a heritable cooperation trait (nature) alongside the learned voluntary join_hunt action (nurture); team-capture success depends on both.

Fixed-trait behavioral & game-theoretic environments

These environments hold every agent trait fixed and instead vary the interaction mechanics or reward shaping. Agents are still born, reproduce, and die, but nothing is inherited or mutated — only the RL policy adapts, converging on a behavioral equilibrium (cooperate, defect, share, reciprocate) under a given incentive design.

• Base environment: the two-policy base environment. Only reproduction rewards. (results)

• Centralized training: a single shared policy across predators and prey, otherwise the base environment.

• Walls occlusion: an extension with walls and occluded vision. Only reproduction rewards.

• Drive-conditioned environment: starts as a copy of the base environment; work in progress toward drive-conditioned behavior.

• Reproduction kick back rewards: on top of direct reproduction rewards, agents receive indirect rewards when their children reproduce.

• Lineage rewards: successor to kick-back rewards; agents are rewarded for descendants surviving over time, with fertility-age caps that shift agents from reproducing to protecting offspring late in life.

• Direct reciprocity: every prey is solo-catchable; predators get a voluntary share_food action, testing whether costly food sharing emerges without any coordination necessity.

• Network reciprocity: reciprocity/sharing decisions are structured over an explicit social-network graph rather than spatial proximity.

• Shared prey: this environment is very similar in logic to mammoth hunting, but in this case the typical energy level of a prey is smaller than that of a predator. With mammoth hunting this is typically the other way around: prey possess more energy than predators. Only reproduction rewards.

• Mammoth hunting: mammoths are only hunted down and eaten by humans in its Moore neighborhood if the cumulative energy of the surrounding humans is strictly larger than the mammoth's energy. On failure (if cumulative human energy is too low), humans optionally lose energy proportional to their share of the attacking group's energy (energy_percentage_loss_per_failed_attacked_prey). On success, prey energy is split among attackers (proportional by default, optional equal split via team_capture_equal_split). Only reproduction rewards. (implementation)

• Mammoths defection: adds a voluntary join/free-ride decision to mammoth hunting.

• "Stag hunt": cooperative and solo hunting with large (mammoths) and small (rabbits) prey. Hunting mammoths usually provides more energy but also needs cooperation of humans and therefore yields a more uncertain outcome.

• Stag hunt forward view: stag hunt defection with forward-shifted predator observations.

• Stag hunt reputation: adds a per-predator reputation signal (join/defect history) on top of forward-view stag hunt defection, to test conditional cooperation.

• Stag hunt vectorized: a performance refactor of stag hunt (vectorized hot paths); not a new behavioral variant.

• Red Queen: independently configurable competing prey types under a shared, non-mutating predator policy, testing coevolutionary arms-race dynamics between learned policies rather than genomes.

• Malthusian RL: two-timescale Leibo-style Malthusian RL — within-episode PPO learning, plus a between-episode reallocation of each species' population share across spatially isolated islands based on measured fitness.

Experiments:

• Testing the Red Queen Hypothesis in the co-evolutionary setting of (non-mutating) predators and prey (implementation, results)

• Testing the Red Queen Hypothesis in the co-evolutionary setting of mutating predators and prey. (implementation, results)

Hyperparameter tuning

• Hyperparameter tuning base environment - Population-Based Training (Implementation)

Installation of the repository

Editor used: Visual Studio Code 1.107.0 on Linux Mint 22.0 Cinnamon

1. Clone the repository:

   git clone https://github.com/doesburg11/PredPreyGrass.git

2. Open Visual Studio Code and execute:
   • Press ctrl+shift+p
   • Type and choose: "Python: Create Environment..."
   • Choose environment: Conda
   • Choose interpreter: Python 3.11.13 or higher
   • Open a new terminal

   • pip install -e .

3. Install the additional system dependency for Pygame visualization:

   • conda install -y -c conda-forge gcc=14.2.0

Quick start

Run a random policy in a Visual Studio Code terminal:

python ./predpreygrass/non_evolutionary/base_environment/random_policy.py

Pretrained checkpoints and historical training outputs are preserved in the legacy archive rather than shipped in the active source tree.

References

Darwinian vs. Baldwinian evolution

The eco_evolutionary_* environments are built around the interaction loop between genetic evolution (Darwinian) and within-lifetime learning (Baldwinian):

• Baldwin, J. M. (1896). A New Factor in Evolution. The American Naturalist, 30(354), 441–451. — the original statement of the effect: learned behavior can steer which genotypes are favored by selection, without the learned behavior itself being inherited.
• Simpson, G. G. (1953). The Baldwin Effect. Evolution, 7(2), 110–117. — clarifies the effect against Lamarckian misreadings and gives it its modern name.
• Hinton, G. E., & Nowlan, S. J. (1987). How Learning Can Guide Evolution. Complex Systems, 1, 495–502. — the canonical computational demonstration: individual learning smooths a rugged fitness landscape, making an otherwise unlikely genotype reachable by evolutionary search.
• Ackley, D., & Littman, M. (1991). Interactions Between Learning and Evolution. In Artificial Life II, 487–509. — evolving agents that also learn during their lifetime, closest in spirit to this repo's genome-plus-PPO setup.

General

• RLlib: Industry-Grade, Scalable Reinforcement Learning
• Paper Collection of Multi-Agent Reinforcement Learning (MARL)
• Multi-Agent Reinforcement Learning: Foundations and Modern Approaches. Stefano V. Albrecht, Filippos Christianos, and Lukas Schäfer