Choi, Minkyu (Lockheed Martin Corp.), Filter, Max (Lockheed Martin), Alcedo, Kevin (Lockheed Martin), Rosenbluth, David (Lockheed Martin), Walker, Thayne (University of Denver, Lockheed Martin), Ide, Jaime, S. (Yale University, Lockheed Martin)

Soft Actor-Critic with Inhibitory Networks for Retraining UAV Controllers Faster

Scheduled for presentation during the Regular Session "Autonomy" (FrC4), Friday, June 24, 2022, 16:00−16:20, Divona-2

2022 International Conference on Unmanned Aircraft Systems (ICUAS), June 21-24, 2022, Dubrovnik, Croatia

This information is tentative and subject to change. Compiled on April 16, 2024

Abstract

Active research in autonomous unmanned aerial vehicles (UAVs) seeks to combine the agility of Proportional-Integral-Derivative (PID) systems for low-level control with the adaptability of Deep Reinforcement Learning (DRL) to navigate through challenging, non-stationary environments. In the real world, there is often a need to quickly adapt trained DRL agents to more difficult tasks with conflicting rewards. For efficient retraining, the ability to leverage previously learned skills becomes critical. Unfortunately, using traditional DRL algorithms like soft actor critic (SAC) for retraining a policy can lead to catastrophic forgetting of the policy's known skills. In this work, inspired by neuroscience research, we propose a novel approach using SAC with inhibitory networks to allow separate and adaptive state value evaluations, as well as distinct automatic entropy tuning. We validate our method through experiments using a quadcopter in a realistic simulation environment and demonstrate the advantage of retraining. Moreover, we present the superiority of our approach compared to baseline methods with respect to both sample efficiency and cumulative success.