Multimodal interaction

Multimodal interaction provides the user with multiple modes of interacting with a system.

Uses your system voice; voice and rate in settings.

Snapshot

≈ 6 min read

Names in other languages

No names in other languages are held for this entry yet.

Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.

Multimodal interaction provides the user with multiple modes of interacting with a system. A multimodal interface provides several distinct tools for input and output of data. Multimodal human-computer interaction involves natural communication with virtual and physical environments.

It facilitates free and natural communication between users and automated systems, allowing flexible input (speech, handwriting, gestures) and output (speech synthesis, graphics). Multimodal fusion combines inputs from different modalities, addressing ambiguities. Two major groups of multimodal interfaces focus on alternate input methods and combined input/output.

Multiple input modalities enhance usability, benefiting users with impairments. Mobile devices often employ XHTML+Voice for input. Multimodal biometric systems use multiple biometrics to overcome limitations.

Multimodal sentiment analysis involves analyzing text, audio, and visual data for sentiment classification. GPT-4, a multimodal language model, integrates various modalities for improved language understanding. Multimodal output systems present information through visual and auditory cues, using touch and olfaction.

Multimodal fusion integrates information from different modalities, employing recognition-based, decision-based, and hybrid multi-level fusion.…

Multimodal interaction provides the user with multiple modes of interacting with a system. A multimodal interface provides several distinct tools for input and output of data. Multimodal human-computer interaction involves natural communication with virtual and physical environments.

It facilitates free and natural communication between users and automated systems, allowing flexible input (speech, handwriting, gestures) and output (speech synthesis, graphics). Multimodal fusion combines inputs from different modalities, addressing ambiguities. Two major groups of multimodal interfaces focus on alternate input methods and combined input/output.

Multiple input modalities enhance usability, benefiting users with impairments. Mobile devices often employ XHTML+Voice for input. Multimodal biometric systems use multiple biometrics to overcome limitations.

Multimodal sentiment analysis involves analyzing text, audio, and visual data for sentiment classification. GPT-4, a multimodal language model, integrates various modalities for improved language understanding. Multimodal output systems present information through visual and auditory cues, using touch and olfaction.

Multimodal fusion integrates information from different modalities, employing recognition-based, decision-based, and hybrid multi-level fusion. Ambiguities in multimodal input are addressed through prevention, a-posterior resolution, and approximation resolution methods.

Multimodal fusion

The process of integrating information from various input modalities and combining them into a complete command is referred as multimodal fusion. In literature, three main approaches to the fusion process have been proposed, according to the main architectural levels (recognition and decision) at which the fusion of the input signals can be performed: recognition-based, decision-based, and hybrid multi-level fusion.

The recognition-based fusion (also known as early fusion) consists in merging the outcomes of each modal recognizer by using integration mechanisms, such as, for example, statistical integration techniques, agent theory, hidden Markov models, artificial neural networks, etc. Examples of recognition-based fusion strategies are action frame, input vectors and slots. The decision-based fusion (also known as late fusion) merges the semantic information that are extracted by using specific dialogue-driven fusion procedures to yield the complete interpretation.

Examples of decision-based fusion strategies are typed feature structures, melting pots, semantic frames, and time-stamped lattices. The potential applications for multimodal fusion include learning environments, consumer relations, security/surveillance, computer animation, etc. Individually, modes are easily defined, but difficulty arises in having technology consider them a combined fusion.

It is difficult for the algorithms to factor in dimensionality; there exist variables outside of current computation abilities. For example, semantic meaning: two sentences could have the same lexical meaning but different emotional information. In the hybrid multi-level fusion, the integration of input modalities is distributed among the recognition and decision levels.

The hybrid multi-level fusion includes the following three methodologies: finite-state transducers, multimodal grammars and dialogue moves.

Ambiguity

User's actions or commands produce multimodal inputs (multimodal message ), which have to be interpreted by the system. The multimodal message is the medium that enables communication between users and multimodal systems.

It is obtained by merging information that are conveyed via several modalities by considering the different types of cooperation between several modalities, the time relationships among the involved modalities and the relationships between chunks of information connected with these modalities. The natural mapping between the multimodal input, which is provided by several interaction modalities (visual and auditory channel and sense of touch), and information and tasks imply to manage the typical problems of human-human communication, such as ambiguity.

An ambiguity arises when more than one interpretation of input is possible. A multimodal ambiguity arises both, if an element, which is provided by one modality, has more than one interpretation (i.e. ambiguities are propagated at the multimodal level), and/or if elements, connected with each modality, are univocally interpreted, but information referred to different modalities are incoherent at the syntactic or the semantic level (i.e. a multimodal sentence having different meanings or different syntactic structure).

In "The Management of Ambiguities", the methods for solving ambiguities and for providing the correct interpretation of the user's input are organized in three main classes: prevention, a-posterior resolution and approximation resolution methods. Prevention methods impose users to follow predefined interaction behaviour according to a set of transitions between different allowed states of the interaction process.

Example of prevention methods are: procedural method, reduction of the expressive power of the language grammar, improvement of the expressive power of the language grammar. The a-posterior resolution of ambiguities uses mediation approach. Examples of mediation techniques are: repetition, e.g. repetition by modality, granularity of repair and undo, and choice.

The approximation resolution methods do not require any user involvement in the disambiguation process. They can all require the use of some theories, such as fuzzy logic, Markov random field, Bayesian networks and hidden Markov models.

See also

Device independence Multimodal biometric system Multimodal search Speech recognition W3C's Multimodal Interaction Activity – an initiative from W3C aiming to provide means (mostly XML) to support Multimodal Interaction scenarios on the Web. Web accessibility Wired glove XHTML+Voice

W3C Multimodal Interaction Activity XHTML+Voice Profile 1.0, W3C Note 21 December 2001 Hoste, Lode, Dumas, Bruno and Signer, Beat: Mudra: A Unified Multimodal Interaction Framework, In Proceedings of the 13th International Conference on Multimodal Interaction (ICMI 2011), Alicante, Spain, November 2011. Toselli, Alejandro Héctor, Vidal, Enrique, Casacuberta, Francisco: Multimodal Interactive Pattern Recognition and Applications, Springer, 2011.

Text derived from https://en.wikipedia.org/w/index.php?oldid=1350803763 under CC BY-SA 4.0; modifications have been made.

Sources cited in this revision

34 named source(s) extracted from the cited revision. A source with no author is shown by its publication; nothing is invented.