During an emergency, the lack of information (provided by IT systems) and therefore situational awareness is one factor that leads to human error and disorganised emergency response. However, unfamiliar locations and uncontrolled conditions are not the most suitable circumstances for deploying extensive IT solutions, especially with high mobility requirements as those of First Responders (FRs).

Moreover, FRs often have tasks that require full attention and the use of both hands to be effective. This severely limits the type of devices they can bring with them and use. Smartphones are the first attempts at providing Situational Awareness to FRs, and special cases have been designed to enable practitioners to use these devices while wearing them around their wrists, freeing their hands. Nevertheless, this still represents an uncomfortable interaction with the system. AR is the next step in providing Situational Awareness and operational assistance to field units, as it offers new possibilities in terms of information display and contextual communication. AR headsets are superior to smartphones for critical information notifications as they can directly capture the user’s attention, even in complex situations. In addition, proper AR glasses can enable true hands-free operations and legible display of information without any action from the practitioner.

In INGENIOUS, CS GROUP studies and develops the Augmented Reality Response Platform (ARRP) for first responders. We research the best compromises as well as user experience models to get the most out of the latest AR technologies. ARRP is designed to provide contextual information to First Responders field units and offer advanced capabilities such as:

• Obtain the position of other team members and assets,
• Annotate the environment,
• Receive the assigned mission,
• Call for the assistance of a remote operator and receive direct instructions,
• Send situation reports to the Control Centre (C2).

CS GROUP also studies new Augmented Reality interactions and visualisations metaphors that currently lack clear definitions and evaluations in terms of usability for security use cases, a necessary step to ensure both the effectiveness of the proposed solution and a high acceptance rate.

During the first period of the project, we initiated the research and development work and selected the AR devices that will be used during the project. We introduced the Augmented Reality (AR) to the end users and presented the selected AR devices (the ORA-2, a standard head-up-display device and the HoloLens 2, a mixed reality device allowing more advanced AR features), the capabilities offered by each, and the key features and interactions relevant for the First Responder (FR) uses cases:

• Display of AR notifications, which can be an alert related to an imminent or nearby threat, or a C2 command.
• Real time sharing of geo-localized information (geo-localization of victims or risks) displayed as Holograms (3D models, images, or texts in the environment) and icon on a map.
• Display the position of the FR and its nearby team members as icon on a map.
• Use of voice command and holographic buttons to trigger an input action such as annotating a specific location.

We are now in the second period of the project and we are working towards the release of a prototype connected to the various INGENIOUS components for end users to test the system in a small-scale test in October in Sweden. Among other works, we are researching:

• The use of non-persistent AR menu displayed on demand in the hand. The goal is to provide access to more actions without overloading the FR field of vision.
• The types of holograms concerned and their representation. Identified and placed from the C2 (on the map) or from FR in the field. The current types are:
  • risks (electrical, fall, biological…) and victim,
  • switch (electrical, gas, water),
  • equipment,
  • marking on door/building.
• The display of up-to-date triage information in AR. Using the AR glasses camera to scan the QR code of the triage device hands-free.
• The remote assistance component and what would be possible to do with it (Access FR video stream, hands-free annotation of the FR view…).

CS GROUP’s Simulation & eXtended Reality Unit is a business line of the Defense & Security Business Unit. This department is a global leader in the X-Reality domain, which today brings together more than 100 PhDs and engineers specialised in Virtual and Augmented Reality systems and real-time computer graphics. It features world-class expertise and industrial references in the following fields:

• Training and e-learning, serious games, interactive storytelling
• Virtual, Augmented, and eXtended Realities
• Situational awareness, common operational picture, GIS, and digital twins
• Live, virtual, and constructive simulation.

Its commercial development is supported by intense R&D activities. Since its creation, this unit has been involved (or is still involved) as a coordinator or as a partner, in several EC funded projects that have produced most of the products the company is currently commercially exploiting.

Watch the following video that summarises the features of the INGENIOUS Augmented Reality Response Platform (ARRP), as tested and validated during Small Scale Test #1 (SST#1).