Memory-Augmented Reinforcement Learning In 3D Space

(2023 National Tsing Hua University CS Department Project Contest Champion out of 94 entries!!!)

In this project, we propose a new episodic memory framework to enhance the ability of the reinforcement learning model to acquire spatial information in a partially observed 3D environment, thereby improving the model's capability to execute tasks within such an environment.

Performance (Average score on collecting apples in 100 different mazes)

Baseline (DQN)	Ours approach
8.4 +/- 6.3	10.3 +/- 9.1

Demo

Baseline (DQN)	Ours approach

Architecture

Installation on Linux

Install DeepMind Lab

Install Pytorch

Install required packages

pip install numpy Pillow opencv-python matplotlib 

Create Task Environment

Create explore_object_locations_xs.lua with the above spec in DeepMind Lab game_script/levels subfolder.

local factory = require 'factories.explore.object_locations_factory'

return factory.createLevelApi{
    episodeLengthSeconds = 135,
    mazeHeight = 9,
    mazeWidth = 9,
    roomCount = 2,
    roomMaxSize = 3,
    roomMinSize = 3,
    objectCount = 8,
    spawnCount = 2,
}

Install level

pip install dm-env

bazel build -c opt --python_version=PY3 //python/pip_package:build_pip_package

./bazel-bin/python/pip_package/build_pip_package /tmp/dmlab_pkg

pip install /tmp/dmlab_pkg/deepmind_lab-1.0-py3-none-any.whl

Training

# Train our proposed reinforcement learning model.
python run_emdqn.py

Testing

# Test our proposed reinforcement learning model in 100 different maze benchmarks.
python run_emdqn.py --test --exp_name {exp name after training} --step {checkpoint step}