Audiovisual prosody in interaction
Research on how voice and face contribute to feedback, turn-taking, communication problems and emotions.
Open publicationAgent interaction research
The research behind conversational agent interactions
Conversational agents can respond not only to the words someone uses, but also to how an interaction develops. Research on audiovisual prosody, spoken feedback, turn-taking, communication problems and emotion provides a framework for recognising moments of understanding, hesitation, emphasis, repair and completion.
The research collected here connects interaction signals to reflective listening, contextual conversational AI and distributed cognition. Together, these perspectives help explain how agents across Agent Interaction Lab can adapt their wording, pace and next step to different roles while keeping observations tentative, transparent and focused on communication rather than diagnosis.
Research Basis
Audiovisual prosody in interaction informs the conversational agents
These publications are relevant across the conversational agents because they examine how people signal understanding, confusion, emotion, turn completion and communication trouble.
Problem detection in human-machine interactions based on facial expressions of users
Research into audiovisual cues to communication problems between users and a spoken dialogue system.
Open publicationThe interplay between the auditory and visual modality for end-of-utterance detection
Research connected to whether someone has finished speaking or still wants to continue.
Open publicationCrossmodal and incremental perception of audiovisual cues to emotional speech
Research connecting agent emotion adaptation to how emotional meaning is expressed across voice and visual cues.
Open publicationEmotion References
Emotion and affect sources used as supporting layers
These sources add an extra affect layer to the interaction model: a compact way to map words and paraphrases to emotion families, tone choices and conversational responses.
Listen, reflect, then ask
The conversational agents use person-centred micro-skills adapted from Rogerian listening and motivational interviewing: open questions, affirmations, concise reflections and summaries. A direct question is answered first. A reflection is phrased as a tentative interpretation that the user can correct, not as a diagnosis or a claim about hidden feelings.
The conversational agents favour a relevant reflection or summary before another question. Each agent applies these skills to its own role: communication, supportive exploration, idea development, requirements or the need behind a design concept. These are communication skills, not therapy.
Contextual Conversational AI
Contextualising conversational AI shapes how agents respond
Lea Krause's thesis Contextualising Conversational AI shapes the next step for the conversational agents across Agent Interaction Lab. It studies how context, indirect signals that someone needs a clearer explanation, and honest uncertainty make a reply more helpful. Across different agent roles, this means noticing when a question, correction or hesitation asks for a shorter, simpler or example-based explanation, while staying careful when information is uncertain — without turning those signals into a lasting profile.
Your thinking, plus the tools you think with
Distributed cognition, from cognitive psychology, argues that thinking is not confined to one mind: it happens across a person and the tools, artefacts and other people around them. Your intelligence in the moment is not just what is in your head, it is you together with whatever you are thinking with.
Agent Interaction Lab is built on that idea at the level of the whole site, not just inside one agent. The user is always part of the loop: a transcript that gets reasoned over, a live analysis panel that reflects the conversation back, agents that hand a partial result to another agent or back to the user. The Idea Sharper pair and the Multi-Agent Solution Designer are one visible instance of this, where the reasoning is split across two agent roles instead of held inside a single one, but the same principle runs through every agent that reasons with the user's own words, transcript and context rather than in isolation.