Catálogo de publicaciones - libros

SmartKom -Home demonstrates the use and benefit of an intelligent multimodal interface when controlling entertainment devices like a TV, a recorder, and a jukebox, and when accessing entertainment services like an electronic program guide combining speech and a handheld display with touch input. One important point is emphasizing the functional aspect, i.e., the user’s needs, conveyed to the system in a natural way by speech and gesture, are satisfied. The user does not need to know device-specific features or service idiosyncrasies. The function modeling component in SmartKom -Home has the necessary knowledge to transform the abstract user request into device commands and service queries.

This paper presents SmartKom -Mobile, the mobile version of the SmartKom system. SmartKom -Mobile brings together highly advanced user interaction and mobile computing in a novel way and allows for ubiquitous access to multidomain information. SmartKom -Mobile is device-independent and realizes multimodal interaction in cars and on mobile devices such as PDAs. With its siblings, SmartKom -Home and SmartKom -Public, it provides intelligent user interfaces for an extremely broad range of scenarios and environments.

People tend to spend an increasing amount of time in their cars and therefore desire high comfort, safety, and efficiency in that environment. A large variety of electronic devices has been made available to meet these requirements in the vehicle. These electronic devices should allow for speech interaction in order to minimize driver distraction and to maximize driver comfort. This contribution studies the user requirements for potential assistant functionalities operated by speech in the car. The architecture is of the dialogue system is defined based on the user requirements study. Speech dialogues were designed according to state-of-the-art principles of human machine interaction for the functionalities desired by the users. Results and ideas for future work conclude this contribution.

This chapter gives a concise overview of the empirical Wizard-of-Oz recordings done within the SMARTKOM project. We define the abstract specifications of the intended simulated communicative situations, describe the necessary technical setup (including numerous useful practical hints), and finally outline the specifications of the resulting multimodal corpus, which may be obtained from the Bavarian Archive for Speech Signals (BAS).

Do users show emotions and gestures if they interact with a rather intelligent multimodal dialogue system? And if they do, what do the “emotions” and the gestures look like? Are there any features that can be exploited for their automatic detection? And finally, which language do they use when interacting with a multimodal system — does it differ from the usage of language with a monomodal dialogue system that can only understand speech? To answer these questions, data had to be collected, labeled and analyzed. This chapter deals with the second step, the transliteration and the labeling. The three main labeling steps are covered: orthographic transliteration, labeling of user states, labeling of gestures. Each step will be described with theoretical and developmental background, an overview of the label categories, and some practical advice for readers who are themselves in the process of looking for or assembling a coding system. Readers who are interested in using the presented labeling schemes should refer to the cited literature — not all details necessary for actually using the different systems are presented here for reasons of space. For information on the corpus itself, please refer to Schiel and Türk (2006).

There are several characteristics not optimally suited for the user state classification with Wizard-of-Oz (WOZ) data like the nonuniform distribution of emotions in the utterances and the distribution of emotional utterances in speech, facial expression, and gesture. In particular, the fact that most of the data collected in the WOZ experiments are without any emotional expression gives rise to the problem of getting enough representative data for training the classifiers. Because of this problem we collected data in our own database. These data are also relevant for several demonstration sessions, where the functionality of the SmartKom system is shown in accordance with the defined use cases. In the following we first describe the system environment for data collection and then the collected data. At the end we will discuss the tool to demonstrate user states detected in the different modalities.

We present three types of data collections and their experimental paradigms. The resulting data were employed to conduct a number of annotation experiments, create evaluation gold standards and train statistical models. The data, experiments and their analyses highlight the importance of data-driven empirical laboratory and field work for research on intuitive multimodal human-computer interfaces.

In this chapter we will give a brief overview about what different methods of evaluation were applied to the SmartKom prototypes and which of them resulted in utilizable results and why others did not. Since there are no established benchmarking methods yet for multimodal HCI systems and very few debatable methods for monomodal dialogue systems our work within the SmartKom prototype evaluations was rather an exploration of new methods than the simple application of standard routines.