TCRP: A Novel Heuristic for Experience Prioritization in Reinforcement Learning

Overview

In reinforcement learning, prioritizing experiences significantly impacts an agent’s learning efficiency and performance. Transition Complexity and Reward Priority (TCRP) algorithm is a novel prioritization strategy that integrates transition complexity with reward magnitude to optimize experience replay. TCRP calculates transition complexity using a state-space metric (e.g., Euclidean distance) to capture environmental changes and combines it with rewardbased prioritization to create a unified priority score. Unlike traditional methods such as Proportional Prioritized Experience Replay (PER), TCRP adapts to dynamic environments by leveraging both structural transitions and reward-driven insights. Experimental results on standard benchmarks like CartPole demonstrate that TCRP accelerates learning convergence, improves policy robustness, and achieves higher average rewards compared to baseline approaches. Furthermore, TCRP’s inherent flexibility makes it suitable for diverse reinforcement learning tasks, offering a promising advancement in efficient memory prioritization.

Features

• Dynamic Experience Prioritization: Combines transition complexity and reward magnitude for better replay efficiency.
• Multi-Environment Support: Works with CartPole, LunarLander, BipedalWalker, and CarRacing environments.
• Deep Reinforcement Learning Integration: Implements Double Deep Q-Networks (DDQN) for better performance.

Repository Structure

This repository contains the following files and directories:

• huber_loss.py - Custom Huber loss function for robust optimization.
• sumTree.py - SumTree structure for prioritized replay buffer.
• main.py - Main script to run the TCRP algorithm.
• dqn_cartpole.ipynb - Jupyter notebook for the CartPole environment.
• dqn_lunar.ipynb - Jupyter notebook for the LunarLander environment.
• dqn_carRacing.ipynb - Jupyter notebook for the CarRacing environment.
• dqn_Bipedal.ipynb - Jupyter notebook for the BipedalWalker environment.
• cartPole_best_model.h5 - Trained model for CartPole.
• logs/ - Directory for TensorBoard logs to monitor training.
• saved_models/ - Directory for storing saved model files.

Getting Started

Installation

1. Install the required dependencies:

    pip install -r requirements.txt

2. Clone the repository:

   git clone https://github.com/nehlinshanila/TCRP-A-Novel-Heuristic-for-Experience-Prioritization-in-Reinforcement-Learning.git
   cd TCRP-A-Novel-Heuristic-for-Experience-Prioritization-in-Reinforcement-Learning
   cd adaptive behaviour algo
   
   

Environments

• CartPole
• LunarLander
• BipedalWalker
• CarRacing

Experimental Setup

The following table outlines the key configurations used:

Configuration	Value
Learning Rate	0.001
Discount Factor	0.95
Replay Buffer Size	100,000
Batch Size	32
Transition Complexity Coefficient	0.00001
Exploration Rate	0.995
Reward Prioritization Exponent	0.8
Total Episodes	2000
Timesteps per Episode	500

Transition Complexity:

Transition Complexity = √Σ_i=1ⁿ (S_1,i - S_2,i)²

priority = (Complexity + ε) · (reward + ε)^α

• S₁ = Current State, S₂ = Next State
• ε = Small value to avoid Zero,
• ε = α = Learning Rate

TCRP Performance Highlights

• Training Time Efficiency: TCRP significantly outperforms DPER and PER in training efficiency, achieving statistical significance with p < 0.01, enabling quicker achievement of performance levels with fewer computational resources.
• Higher Average Rewards: Demonstrates significant improvement in average rewards over DPER and PER, with a statistical significance of p < 0.05, showing that TCRP not only learns faster but secures better long-term reward outcomes.
• Faster Training Convergence: TCRP requires fewer episodes to reach optimal performance levels, indicating a faster convergence speed, highlighting its efficiency in leveraging high-priority transitions for quicker learning progress.
• Robust Across Metrics: TCRP shows consistent superiority in training efficiency, reward gains, and convergence speed, emphasizing its robustness and effectiveness.
• Effective in Dynamic Environments: Particularly potent in environments with dynamic state transitions, surpassing traditional methods in identifying and leveraging meaningful experiences.
• Scalability: Maintains performance advantages without significant computational overhead as buffer capacities and dataset sizes increase.

License

This project is licensed under the MIT License.