Augmented Reality for Live Fire Ranges SBIR

BANC3, under a Phase I SBIR for PEO-STRI, designed an Augmented Reality (AR) system for basic marksmanship training on live fire ranges. This challenging research project's design and development reinforced BANC3's understanding of applying AR in a military environment, as well as designing an AR application for use in real-time. BANC3 is continuing development on this project after being selected for a Phase II award.

The soldier wears an optical–see–through helmet mounted display (HMD) that has integrated sensors such as cameras, an Inertial Measurement Unit (IMU), and depth and other sensors. The HMD is connected to a small computer worn unobtrusively, either on the back of the helmet or on the body. When the soldier has the rifle down; the HMD displays the virtual targets downrange in both eyes, superimposed over the real world. When the soldier raises the rifle to aim, the HMD detects and tracks the rifle and any sights, and adjusts the view of the virtual objects for the eye used for aiming to match the magnification and obscuration of that particular optic. The virtual targets are displayed in the eyepieces matching the correct aspect ratio, size, position, orientation, stability, illumination, and shading to give the illusion that the object is placed on the ground in a fixed location. To further aid in the realism, a pre-loaded terrain map is used to accurately determine if the target should be partially obscured by terrain, and is drawn accordingly. Thus if the target is partially behind a berm, only the visible part will be shown on the HMD.

The sensors in the HMD are also used to detect and track nearby occluders such as hands, rifle, sand bags, or any other shooting platform. These detections will be used to correctly occlude the virtual objects being displayed on the HMD, such that they appear correctly to be behind them. The system also needs to know whether the rifle is equipped with an optic sight, what type it is, and where it is located with respect to the eye of the soldier, in order to be able to correctly display the virtual object as if appearing through the optic itself. The accuracy and alignment of this detection is crucial to maintaining realism and ensuring constructive training. Sensors in the HMD detect and track the sight type and location. Once the soldier fires a shot, LOMAH systems down range will detect the exact shot placement and relay that information back to the AR system so that the virtual object can be impacted accordingly. The AR system can provide instantaneous feedback to the shooter about shot placement, and can even show a "zoomed in view" of the virtual target to highlight where the virtual impact has occurred.