CSC News

New AI Boosts Teamwork Training

For Immediate Release

Wookhee Min | wmin@ncsu.edu

Jay Pande | jpande@ncsu.edu

James Lester | lester@ncsu.edu

Matt Shipman | matt_shipman@ncsu.edu

Researchers have developed a new artificial intelligence (AI) framework that is better than previous technologies at analyzing and categorizing dialogue between individuals, with the goal of improving team training technologies. The framework will enable training technologies to better understand how well individuals are coordinating with one another and working as part of a team.

“There is a great deal of interest in developing AI-powered training technologies that can understand teamwork dynamics and modify their training to foster improved collaboration among team members,” says Wookhee Min, co-author of a paper on the work and a research scientist at North Carolina State University. “However, previous AI architectures have struggled to accurately assess the content of what team members are sharing with each other when they communicate.”

“We’ve developed a new framework that significantly improves the ability of AI to analyze communication between team members,” says Jay Pande, first author of the paper and a Ph.D. student at NC State. “This is a significant step forward for the development of adaptive training technologies that aim to facilitate effective team communication and collaboration.”

The new AI framework builds on a powerful deep learning model that was trained on a large, text-based language dataset. This model, called the Text-to-Text Transfer Transformer (T5), was then customized using data collected during squad-level training exercises conducted by the U.S. Army.

“We modified the T5 model to use contextual features of the team – such as the speaker’s role – to more accurately analyze team communication,” Min says. “That context can be important. For example, something a team leader says may need to be viewed differently than something another team member says.”

To test the performance of the new framework, the researchers compared it to two previous AI technologies. Specifically, the researchers tested the ability of all three AI technologies to understand the dialogue within a squad of six soldiers during a training exercise.

The AI framework was tasked with two things: classify what sort of dialogue was taking place, and follow the flow of information within the squad. Classifying the dialogue refers to determining the purpose of what was being said. For example, was someone requesting information, providing information, or issuing a command? Following the flow of information refers to how information was being shared within the team. For example, was information being passed up or down the chain of command?

“We found that the new framework performed substantially better than the previous AI technologies,” Pande says.

“One of the things that was particularly promising was that we trained our framework using data from one training mission, but tested the model’s performance using data from a different training mission,” Min says. “And the boost in performance over the previous AI models was notable – even though we were testing the model in a new set of circumstances.”

The researchers also note that they were able to achieve these results using a relatively small version of the T5 model. That’s important, because it means that they can get analysis in fractions of a second without a supercomputer.

“One next step for this work includes exploring the extent to which the new framework can be applied to a variety of other training scenarios,” Pande says.

“We tested the new framework with training data that was transcribed from audio files into text by humans,” Min says. “Another next step will involve integrating the framework with an AI model that transcribes audio data into text, so that we can assess the ability of this technology to analyze team communication data in real time. This will likely involve improving the framework’s ability to deal with noises and errors as the AI transcribes audio data.”

The paper, “Robust Team Communication Analytics with Transformer-Based Dialogue Modeling,” will be presented at the 24th International Conference on Artificial Intelligence in Education (AIED 2023), which will be held July 3-7 in Tokyo, Japan. The paper was co-authored by Jason Saville, a former graduate student at NC State; James Lester, the Goodnight Distinguished University Professor in Artificial Intelligence and Machine Learning at NC State; and Randall Spain of the U.S. Army Combat Capabilities Development Command (DEVCOM). Soldier Center.

This research was sponsored by the U.S. Army DEVCOM, Soldier Center under cooperative agreement W912CG-19-2-0001.

-shipman-

Note to Editors: The study abstract follows.

“Robust Team Communication Analytics with Transformer-Based Dialogue Modeling”

Authors: Jay Pande, Wookhee Min, Jason D. Saville and James Lester, North Carolina State University; Randall Spain, U.S. Army Combat Capabilities Development Command

Presented: July 3-7, AIED 2023, Tokyo, Japan

Abstract: Adaptive training environments that can provide reliable insight into team communication offer great potential for team training and assessment. However, traditional techniques that enable meaningful analysis of team communication such as human transcription and speech classification are especially resource-intensive without machine assistance. Additionally, developing computational models that can perform robust team communication analytics based on small datasets poses significant challenges. We present a transformer-based team communication analysis framework that classifies each team member utterance according to dialogue act and the type of information flow exhibited. The framework utilizes domain-specific transfer learning of transformer-based language models pre-trained with large-scale external data and a prompt engineering method that represents both speaker utterances and speaker roles. Results from our evaluation of team communication data collected from live team training exercises suggest the transformer-based framework fine-tuned with team communication data significantly outperforms state-of-the-art models on both dialogue act recognition and information flow classification and additionally demonstrates improved domain-transfer capabilities.