Multimodal Software Engineering

Research focus

The mission of the Multimodal Software Engineering group is to provide methods and technologies to enable human-centric, intuitive and “natural” interaction between humans and computing systems everywhere. The idea of ubiquitous computing requires re-thinking of existing GUI approaches: users have more than one device to interact with, but these devices vary strongly regarding their input and output capabilities (different screen sizes, voice recognition, etc.). Intelligent user interfaces will adapt themselves to the current situation using context information in order to proactively support each individual user in an optimal way.

Multimodal software engineering has to deal with many research challenges for this vision to become true:

How to use other forms than point-and-click interfaces, like multi-touch displays, voice input and speech output?
How to integrate context information to build intelligent user interfaces and facilitate human computer interaction?
How to develop distributed user interfaces combining various interaction modalities?
How to modify existing software engineering processes in order to account for multimodal applications?

Meeting these challenges a number of modern software engineering approaches and software technologies are combined with concepts from computational intelligence and human-computer interaction. In dealing with these fields the Multimodal Software Engineering group has acquired practical experience with the following topics:

Model-driven software engineering (MDSE)
Service-oriented architectures (SOA)
Design patterns for multimodal user interfaces
Ontologies for context knowledge
Machine learning (case-based reasoning, statistical learning)
Wearable and mobile computing
User interface evaluation

This knowledge is constantly applied and refined in a number of public and industry funded projects performed in such different domains as public security, intelligent cars, process industry, customer relationship management or web-based applications.

Projects

Augur - AUGmenting User interfaces with prediction and modality through Reasoning
Proactive user interfaces facilitate the user interaction by predicting the user's next steps and giving proactive support, e.g. by highlighting the next step or prefilling input fields. Context information from various sources is combined and used to reason about the task the user is trying to achieve.
SoKNOS - Service-oriented architectures to support networks for public security
The SoKNOS project develops and evaluates concepts to support govermental agencies, organizations and companies to provide public security by enabling ad-hoc information sharing and process integration among all relevant actors in case of emergencies. Special emphasis is put on the human-computer-interaction in emergency scenarios and the use of novel interaction modalities.
emode - Enabling Model Transformation-Based Cost Efficient Adaptive Multi-modal User Interfaces
Emode aims at providing a more efficient software development process for multimodal applications. A model-driven approach is used to generate user interfaces semi-automatically for a variety of different devices and to integrate context information.
Interaction Design of Wearable Computing for Mobile Workers
The design of wearable computing solutions for mobile workers requires and in-depth understanding of the tasks they have to perfom. Therefore, tools and models to observe the task and the available ressources are provided in order to allow a semi-automatic selection of possible interaction devices and strategies.

People

Area Head

Dr. Felix Flentge

Research Staff