Small Scale Tests of the INGENIOUS K9 Vest, MAX, MINs, IPS, MACS-SaR, Field Communication components (SST#4 & SST#5)
Europe remains vulnerable to a wide range of natural and man-made hazards (such as floods, earthquakes, industrial accidents, loss of critical infrastructure, etc.), which have a huge impact on human lives and economic losses.
INGENIOUS is a project created to respond to all these threats, through the creation of tools capable of ensuring the self-protection of first responders; offering a coordinated response extensively validated by European and international researchers and technology developers; developing a set of portable technologies and miniature sensors to protect and empower first responders; developing a range of portable devices and rapidly deployable platforms in the field that gather information and optimise communication between responders and victims; merging all information within a single framework and developing an advanced C3 (Command, Control & Coordination) that provides a common operational picture for all responders; conveying a common operational picture to first responders by exploiting the latest advances in augmented reality and application development to increase coordination; enhancing public safety by increasing the capabilities of first responders; enabling cross-border collaboration by overcoming language and bureaucratic barriers through technology; ensuring legal, social, ethical and security considerations; and engaging all technology participants and industry professionals towards maximum market penetration and exploitation of the toolkit.
With the aim of testing the tools and technologies developed within the project in the field, the so-called SSTs (Small Scale Tests) were held on 5 and 6 October 2021 at the Berrozi (Basque Country) facilities, in Bilbao, Spain, organised by our Ertzaintza partners. Specifically, these days were used to demonstrate the SST#4 and SST#5, and were attended by the partners ICCS, ITC, Tekniker, SBFF, FOI, SINTEF, DLR and of course the Ertzaintza, which had experts from different units who were able to see and evaluate the tools.
The first part of the SSTs consisted of a briefing at the Ertzaintza facilities in Iurreta, where ICCS (Project Coordinator) and the Ertzaintza European Projects team presented the project and the agenda for the two days both to the members of the department present and to the project partners.
Once on the field, they began with the distribution of tasks around the Berrozi facilities and the scenarios where the tests would be carried out were differentiated.
SST#4 – INGENIOUS K9 Vest Testing
The SST#4 (K9 Vest) is intended to increase the operational capability of the K9 unit, the situational awareness of the K9 trainer and provide bidirectional communication with the command center. Currently, K9 units operate without any technological equipment, relying heavily on the training of the dog and the collaboration between dog and handler. The INGENIOUS K9 Vest provides situational awareness from the dog’s location (GPS), two video streams (HD and thermal camera) and bi-directional audio between the dog and the handler/command center.
Localization, video streaming and bi-directional audio were tested in a location including a forested area with natural and artificial obstacles (briars, car parts, ironwork, pieces of masonry) as well as an enclosed area (small building). Several rescue scenarios were tested, both in a forested area and in the enclosed area. The vest was also stress tested with the K9 crossing an area with low and narrow obstacles several times and during running. The dog’s comfort while wearing the vest was extensively evaluated by the handlers during the two days of the SST.
During the interaction of the dogs, part of the Ertzaintza canine unit with the vest and the supervision of the instructors and technological partners, the capabilities of the device could be analysed, as well as the failures and needs for improvement for a total adaptation in the field.
On the other hand, the SST#5 consisted of testing mapping and self-localization of first responders (FRs) using drones and the Integrated Positioning System (IPS), both indoor and outdoor. Indoor drones are drones that support the location of FRs in semi-collapsed and dangerous buildings. Currently, GPS signal or similar technologies do not work in indoor environments, so the team working outside does not know the position of the FRs operating inside the building. Drones attempt to solve this problem by creating a network of self-deployable drones to track the movement of FRs while operating indoors.
At the same time, the outdoor drone or Modular Aerial Camera System – Search and Rescue (MACS-SaR) consists of a VTOL drone and a special camera system that generates and provides a map of a region of interest and its surroundings for a quick assessment of the disaster. Also, Field Communication deployed by ICCS to build a stable field communication system to support onsite data transmission.
The Ground Control Station is a data processing center and a transmission interface. All the tools in the SST#5 are connected to the Ground Control Station by using a same communication protocol. The Ground Control Station is able to receive data from one tool and send it to another.
SST#5 – INGENIOUS MINs Testing
More specifically, our partners from SINTEF, Norway, are responsible for the deployment of the small aerial vehicles, so-called MINs – Micro Indoor drones, and for the formation of the aforementioned mesh. The tests in Bilbao allowed the INGENIOUS partners to test new features and gave valuable insights. For example, they were able to perform a test with 10 MINs flying up to 15 meters. This is a scale that the SINTEF team seldomly reaches in its own premises due to a lack of space and safety. While flying, the MINs successfully avoided collisions with obstacles due to SINTEF’s newly developed obstacle avoidance algorithm. Next to testing and development, an important part of such SSTs is integration with partners. Thus, the SSTs gave the opportunity to our SINTEF partners to test the integration with our colleagues from FOI among others. Their drone mapped the environment and provided this map to SINTEF. They were then able to define the flight path for the MINs in a smarter way as without such a map.
The tests also visualized potential areas for improvement. The large amount of MINs was a severe test for the communication protocol. Thus, the next steps include investigating more efficient communication in the future in order to allow for even larger swarms. Also, quick tests on the ranging capabilities of SINTEF’s UWB devices were conducted. While these were interesting, our partners realized they have to do a more solid, quantitative analysis back in Norway. That way they can optimize the usage of the MINs (fewer MINs for covering larger areas) and have better control over the localization accuracy of the first responders.
SST#5 – INGENIOUS MAX drone & AI algorithms Testing
The second type of drone that is being developed within INGENIOUS is the Multi-purpose Autonomous eXploring drone, MAX drone, which is an autonomous drone for unsupervised exploration and assessment of high-risk environments. At SST#5 our partners from FOI demonstrated the MAX drone’s ability to fly and navigate autonomously in an indoor environment. Algorithms and components that had until then been used separately were integrated on the same platform and run in parallel in real-time for the purposes of the SST.
Images collected with the MAX cameras were sent to the Ground Control Station, showing the ability to use MAX as a scout that sends data from within the building in real-time. Researchers from ITC designed artificial intelligence (AI) algorithms that were able to analyze the received images and output high level semantic information. One of the algorithm was designed for low-light environments and it could represent an image as a semantic map and denote important elements such as doors and windows. The ITC team also demonstrated their victim detection algorithm using deep learning technology, which was designed for detecting victims buried in debris. At the initial stage of algorithm design, they considered the lack of real training data, so they used composite images for training. The algorithm had achieved good performance.
A 3D data set collected with the drone’s onboard laser scanner was shared with the MIN drone systems. The collected 3D point cloud shows the geometry/layout of the building and helps a user deploy MIN drones efficiently.
The SST was a good opportunity for our partners to test the effectiveness of their tools and identify challenges and gaps in further development. Hence, some instability in the platform’s flying behavior was noticed, that produced a “swaying” behavior, that in turn prevented our partners from performing too bold flying exercises as MAX needed to stay safely away from obstacles.
In addition, at SST#5 MAX did not have obstacle avoidance functionality on-board, which made it necessary to put restrictions on the movement of the platform, to be sure that MAX did not make follow a computed path into to an obstacle.
SST#5 – INGENIOUS deployable Integrated Positioning System (IPS) & Modular Aerial Camera System – Search and Rescue (MACS-SaR) Testing
During SST#5 two more of the INGENIOUS systems both developed by our DLR partners were tested and validated. The deployable Integrated Positioning System (IPS) supports the self-localisation of FRs, offering seamless navigation in indoor and outdoor environments, while it also formulates a map of the surrounding environment, detecting and tracking persons and assets in real time. The Modular Aerial Camera System – Search and Rescue (MACS-SaR) provides rapid aerial mapping, generating georeferenced data for an instantaneous overview of the area as a large scale situational picture (LS).
SST#5 allowed the DLR partners to demonstrate the technologies and test their features and integration with the other INGENIOUS components. The self-localization test with IPS was conducted online in a global reference frame (world coordinate system) by means of using April Tag coordinates, which have been provided by MACS rapid aerial mapping short before the IPS measurements. The IPS online self-localization data generated were successfully streamed over the INGENIOUS server.
However, testing of the IPS had also some difficulties. The high software complexity led to system instabilities, especially temporal Wi-Fi connectivity failures to the INGENIOUS server caused IPS software crashes. The test also outlined the needs and potentials for optimization of IPS – HW/SW-components for the application.
MACS-SaR testing was also successful. During the MACS-SaR flight large scale operational pictures were generated and delivered as planned. Feature points (world coordinate system) to co-register all other components tested in Bilbao were also successful delivered.
The test was productive in all respects, as in addition to testing and validating the technologies, areas for improvement were identified at the same time. For example, DLR partners came to the conclusion that degree of automatization to extract feature points (April-/QR-Tags) needs to be enhanced. In addition, WiFi transmission test to the INGENIOUS GCS is an open issue and could not be tested during SST#5.
Ertzaintza was in charge of organising these Testing activities as a commitment acquired in the first design phase of the project, and also played a fundamental role in ensuring that the development of the tools was adapted to the real needs of the FRs due to Ertzaintza’s extensive experience in critical incident response and management.
For the evaluation of the tools tested in the SSTs, the Ertzaintza deployed in the field the entire holistic range of units involved in this type of incident, which were represented by one or more of their members, with the intention of providing their expert knowledge and working together with the technological partners. These units, coordinated by the European Projects team, included the Canine Unit, the Helicopter and Drone Unit, SWAT, the Mobile Brigade Unit, the Mountain Rescue Unit, the Marine Rescue Unit, mapping experts and members of the Training Department.