Natural Language Processing (TALN) Group at the UPF has been working within the INGENIOUS project on the Multilingual Operations App (MOA) which aims to facilitate the communication between international First Responder teams (FRs) and the Command Center through the COP/C3 platform. Specifically, the MOA does speech-to-speech translation, enabling the FRs to send voice messages in their own language that will be transcribed, translated to English and received in the COP/C3 as text and/or voice messages.

The Multilingual Operations App is composed of four different modules:

• The Automatic Speech Recognition (ASR) module, in charge of transcribing the FRs messages to text.
• The Machine Translation (MT) module, in charge of translating the transcribed FRs messages from their source language to English.
• The Speech-To-Speech (TTS) module, in charge of converting the translated text messages into the output audio.
• The app interface, in charge of acting as the facade of the whole system, allows the users to send their messages and receive the translations.

1st phase: LIT #9

The first iteration of the MOA development presented the users a web interface that allowed them to try each of the steps involved in speech-to-speech translation. This web interface relies on a REST interface that exposes an endpoint for each component separately and an endpoint for the whole speech-to-speech process. The first version of the MOA featured only Spanish to English translation, as the starting point for the development, and support for more input languages was added in subsequent phases. This version of the MOA was presented to the INGENIOUS partners during the Laboratory Integration Test #9 (LIT#9) which was held in November 2020.

2nd phase: LIT #18

In the Laboratory Integration Test #18 (LIT #18), which took place virtually on the 21rst of April 2021, the second iteration of development was showcased, featuring the new Telegram bot interface and extended language support (Spanish and French as input language, with translation to English). Using Telegram as the new user interface allowed easy access from any mobile device and provided a useful set of tools for access control, user management and conversation tracking.

This version of the MOA is presented in the following video:

3rd phase

In the next phase, the bot was connected with the COP through the Kafka bus used throughout the Ingenious project for communication between components. This allows operators at the COP to have access to a log of conversations on the field, along with the corresponding transcripts and translations. Management of users and teams is handled by the COP. German and Swedish were added as input languages.

This version was tested during SST #9 in Sweden from 11 to 15 October 2021, with partners from different countries representing all four input languages participating in the test. This was the first opportunity to work in person with end users to test the functionality of the MOA, whereas the prior LITs were more limited remote testing sessions.

Despite the fact that the MOA, being an internet-based communication tool, can technically be tested from remote locations, the direct interaction with end users from different backgrounds was very helpful in getting much more immediate feedback on the potential as well as the current limitations of the Multilingual Operations App.

The reception was very positive, with clear interest in the functionality of the MOA from many partners. The use of Telegram as a prototype platform was also well received, as it makes it easy to use the MOA from a variety of devices. Having to register with the Telegram service is not ideal, but was accepted by all users.

The testing still showed problems with the transcription and translation quality, especially in noisy environments, as well as the importance of being able to quickly review the messages that are sent in order to avoid miscommunications. However, it also showed the potential of the MOA and the interest of the user partners in having such an application.

Next steps

The next steps in the development involve performance improvements of the different components, both in terms of quality and speed and adjustments of the Telegram bot based on the analysis of the user feedback from SST#9.