Multimodal Interaction Technologies to Support Small Unit Leaders

Unmanned systems can provide a wealth of information to the warfighter on the modern battlefield, and there are a multitude of control modalities (such as tablet or computer) available to the operator. The dynamic selection of the correct modality can optimize both mission performance and situational awareness.

Our team developed several new drone control methodologies and demonstrated the transitions between them for small UAVs. We first developed an open Android drone control application using the Parrot Anafi SDK. This application runs on our AR headset (HUD), standard Android phones, and tablets. Then we developed a feature that uses our SLAM/VIO position tracking technology to enable the drone to follow the operator without GPS and without having its camera focused on them.

Next, we developed software to connect to a COTS 360-degree camera that we mounted to the drone. With this feature, an operator can fly a drone into a room while we project a 360-degree sphere of real-time video around them. Now they can survey every angle of the space around the drone simply by looking around themselves with their HUD. Then, we added nudge control, operated via a ring controller, and maneuver via a Bluetooth-connected controller.

Since we were streaming a 360-degree feed, there was no waiting for the drone to turn or a gimble to rotate. The operator could see anywhere around the drone just by moving their head. We also developed our own open drone camera streaming client to easily transition between a heads-up drone flying view in our AR HUD and the 360-sphere view around the drone. A second important technology we implemented was drone follow-me. With follow-me, we used our AR headset's SLAM/VIO position tracking to track the operator's position as they moved through both interior and exterior spaces. Then, we translated these movements into drone commands to follow the operator and created an integrated control system where a drone will move with the operator, while allowing them to have both hands on the rifle, as well as look anywhere around the drone. Such an integrated system allows an operator to move through dangerous areas such as buildings, narrow streets, and blind corners with both hands on their weapon. This approach would leave them ready to engage while at the same time receiving valuable real-time imagery from the drone on potential threats.