CODE 80164 ACADEMIC YEAR 2024/2025 CREDITS 6 cfu anno 2 COMPUTER ENGINEERING 11160 (LM-32) - GENOVA SCIENTIFIC DISCIPLINARY SECTOR ING-INF/05 LANGUAGE English TEACHING LOCATION GENOVA SEMESTER 1° Semester TEACHING MATERIALS AULAWEB OVERVIEW This course provides students with foundational conceptual knowledge, methodologies, and tools for designing, implementing, and evaluating computer systems that can capture, represent, and automatically analyze the behavior of their users and interact with them by generating multisensory feedback in real-time. AIMS AND CONTENT LEARNING OUTCOMES This course provides students with foundational conceptual knowledge, methodologies, and tools for designing, implementing, and evaluating computer systems that can capture, represent, and automatically analyze the behavior of their users (e.g., in terms of gesture, movement, facial expressions, speech) and interact with them by generating multisensory feedback (e.g., images, sounds, control of actuators) in real-time. AIMS AND LEARNING OUTCOMES The course aims at introducing the foundational knowledge needed for designing and developing computer systems that can interact with their users naturally, by exploiting multiple sensory channels. This requires students to know and apply technologies for capturing, representing, and automatically analyzing the behavior of the users – e.g., algorithms for detecting and analyzing gesture, full-body movement, facial expressions, and speech – and for generating multisensory feedback (e.g., images and sounds) in real-time. At the end of the course, students will: Know and understand the motivations for using a multimodal interactive system for a specific application, the logical architectures that describe the major components a multimodal interactive system consists of, the guidelines for designing and developing multimodal interactive systems, the application areas where multimodal interactive systems can be successfully exploited. Know the most relevant devices for capturing data that can characterize the behavior of the users and understand how they work and when and how they can be used. Know the most important techniques for representing and automatically analyzing the behavior of the users and understand how and when to apply them. Techniques receive as input data coming from multiple sensor devices covering multiple sensory channels. Be able to analyze specific use cases in selected application areas to evaluate pros and cons for developing a multimodal interactive system rather than a traditional graphical user interface. Be able to design a multimodal interactive system and to implement its major components, by using the development tools presented during lectures and hands-on in the course. PREREQUISITES It is useful, even if not strictly required, having some basic knowledge on human-machine interaction topics (design, development, and evaluation cycle of traditional user interfaces, interaction design methodologies). It is also useful having a basic programming experience. TEACHING METHODS The course includes theoretical and practical lectures. Theoretical lectures aim at introducing the concepts and the techniques the course focuses on. Practical lectures consist of hands-on and enable students to apply the presented concepts and technologies in specific case studies. Students can attend the practical lectures by using their laptops in the classroom. Practical lectures exploit tools for gesture and movement analysis and for audio processing. Working students and students with certified DSA, disability or other special educational needs are advised to contact the lecturer at the beginning of the course to agree on teaching and examination arrangements that, while respecting the teaching objectives, take into account individual learning patterns. SYLLABUS/CONTENT Introduction to multimodal systems: post-WIMP interfaces, multimodal interfaces, definition of multimodal system. Designing multimodal systems: motivations, guidelines for designing multimodal systems, frameworks. Automated analysis of body movement and gesture: devices for motion capture, techniques for extracting movement and gesture features, movement segmentation, gesture recognition, analysis of expressive gesture. Automated analysis of facial expression: techniques for face detection, techniques for extracting facial landmarks, techniques for extracting facial features, analysis of facial action units. Automatic speech recognition: audio capturing, recording, and playback, techniques for extraction and analysis of audio features (temporal, spectral, and cepstral features), introduction to automatic speech recognition. Multimodal machine learning: methods for multimodal integration, early integration model, late integration model. RECOMMENDED READING/BIBLIOGRAPHY Learning material includes pdf copies of the slides presented during the lectures, examples and exercises for the practical lectures, and a collection of scientific papers the teacher provides the students with. Learning material is made available on AulaWeb. Students that cannot attend lectures as well as students with disabilities or learning disorders can contact the teacher to get further indication about the learning material. TEACHERS AND EXAM BOARD GUALTIERO VOLPE Ricevimento: The lecturer is available by appointment at his office on the top floor of Villa Bonino, Viale Causa 13, 16145, Genoa, or at Casa Paganini, Piazza S. Maria in Passione 34, 16123, Genoa, or remotely on the Microsoft Teams platform. To make an appointment, send an e-mail to gualtiero.volpe@unige.it Exam Board GUALTIERO VOLPE (President) ELEONORA CECCALDI RADOSLAW NIEWIADOMSKI ANTONIO CAMURRI (President Substitute) LESSONS LESSONS START https://easyacademy.unige.it/portalestudenti/index.php?view=easycourse&_lang=it&include=corso Class schedule The timetable for this course is available here: Portale EasyAcademy EXAMS EXAM DESCRIPTION The exam consists of a project and an oral discussion. The project is assigned by the teacher and concerns the design of a multimodal interactive system for a specific application. It may include the development of software modules for audio and video processing aiming at contributing to the analysis of the behavior of the users, a deeper analysis of specific topics presented during the lectures by means of a targeted bibliographical search, the analysis of existing solutions, including testing and assessment of algorithms and their performances. Alternatively, the project may concern the analysis and presentation of a scientifc paper assigned by the teacher. The oral discussion consists of the presentation of the results of the project and also includes questions on topics addressed during the lectures. Students with certification of Specific Learning Disabilities (SLD), disabilities, or other special educational needs must contact the instructor at the beginning of the course to agree on teaching and examination methods that, while respecting the course objectives, take into account individual learning styles and provide appropriate compensatory tools. It is reminded that the request for compensatory/dispensatory measures for exams must be sent to the course instructor, the School representative, and the “Settore servizi per l'inclusione degli studenti con disabilità e con DSA” office (dsa@unige.it) at least 10 working days before the test, as per the guidelines available at the link: https://unige.it/disabilita-dsa. ASSESSMENT METHODS The project aims at assessing the extent to which the student is able to analyze a case study (or a scientific paper), to evaluate the opportunity of developing a multimodal interactive system in such a context, to design a multimodal interactive system, and to apply the technologies presented during the lectures. The oral discussion aims at assessing the extent to which the student knows and understands the foundational theoretical aspects of multimodal systems (motivations, logical architectures, and guidelines for design and development) as well as the student’s understanding of the major data acquisition devices and of the techniques for representation and automated analysis of the behavior of the users. The teacher will assess the quality of the project and of the presentation of the results, the capability of critical reasoning on the developed project, the correct use of specialized lexicon, the deepness of the student’s knowledge and understanding of the topics addressed in the course, and the capability of presenting such a content properly. FURTHER INFORMATION Master theses are available concerning the topics presented in the course and addressing the design and development of multimodal interactive systems in areas of interest for the scientific and technological research carried out at the Casa Paganini – InfoMus research center of DIBRIS – University of Genoa (www.casapaganini.org). For students who are interested in a master thesis on these topics, the course provides the theoretical and practical knowledge, which is needed to carry out the work in the thesis. Agenda 2030 - Sustainable Development Goals Good health and well being Quality education Sustainable cities and communities