DRL Robot navigation in IR-SIM

Deep Reinforcement Learning algorithm implementation for simulated robot navigation in IR-SIM. Using 2D laser sensor data and information about the goal point a robot learns to navigate to a specified point in the environment.

Installation

• Package versioning is managed with poetry
  pip install poetry
• Clone the repository
  git clone https://github.com/reiniscimurs/DRL-robot-navigation.git
• Navigate to the cloned location and install using poetry
  poetry install

Training the model

• Run the training by executing the train.py file
  poetry run python robot_nav/train.py

• To open tensorbord, in a new terminal execute
  tensorboard --logdir runs

Sources

Package	Description	Source
IR-SIM	Light-weight robot simulator	https://github.com/hanruihua/ir-sim
PythonRobotics	Python code collection of robotics algorithms (Path planning)	https://github.com/AtsushiSakai/PythonRobotics

Models

Model	Description	Model Source
TD3	Twin Delayed Deep Deterministic Policy Gradient model	https://github.com/reiniscimurs/DRL-Robot-Navigation-ROS2
SAC	Soft Actor-Critic model	https://github.com/denisyarats/pytorch_sac
PPO	Proximal Policy Optimization model	https://github.com/nikhilbarhate99/PPO-PyTorch
DDPG	Deep Deterministic Policy Gradient model	Updated from TD3
CNNTD3	TD3 model with 1D CNN encoding of laser state	-
RCPG	Recurrent Convolution Policy Gradient - adding recurrence layers (lstm/gru/rnn) to CNNTD3 model	-
MARL: TD3-G2ANet	G2ANet attention encoder for TD3 model in MARL setting	G2ANet adapted from https://github.com/starry-sky6688/MARL-Algorithms
MARL: TD3-IGS	In-Graph Softmax attention model for TD3 model in MARL setting	-

Max Upper Bound Models

Models that support the additional loss of Q values exceeding the maximal possible Q value in the episode. Q values that exceed this upper bound are used to calculate a loss for the model. This helps to control the overestimation of Q values in off-policy actor-critic networks. To enable max upper bound loss set use_max_bound = True when initializing a model.

Model
TD3
DDPG
CNNTD3

MARL Models

Multi agent RL setting with training multiple robots a the same time with a single policy. Implementation does not use sensor informaition and only exchanged graph messages are used for navigation.

Video of results: