AIMS AND CONTENT

LEARNING OUTCOMES

This teaching unit aims to provide students with the theoretical and practical knowledge needed to design, implement, and evaluate autonomous agents capable of learning to interact with unknown environments through reinforcement learning algorithms, while also developing critical thinking and independent learning skills in applied contexts.

AIMS AND LEARNING OUTCOMES

By the end of the course, students will be able to:

• formulate a sequential decision-making problem by identifying the agent, environment, states, actions, rewards and learning objectives;
• explain the main reinforcement learning concepts and algorithms using appropriate technical terminology;
• distinguish between tabular methods and function approximation approaches, discussing their assumptions, advantages and limitations;
• explain the roles of policy evaluation, policy improvement, and exploration and exploitation in the learning process;
• select suitable algorithms for a given problem and justify their choices in relation to the available information, computational resources and task requirements;
• implement and test reinforcement learning algorithms using Python and Jupyter Notebook;
• evaluate agent performance quantitatively, interpreting learning curves and considering variability across experimental runs;
• compare alternative approaches and critically discuss their effectiveness and limitations;
• document implementations and communicate experimental methods, results and conclusions clearly;
• independently use technical documentation and scientific literature to extend their knowledge and address implementation problems.

TEACHING METHODS

The course combines lectures with practical programming sessions. Slides, source code and supporting materials will be available on aul@web.

During lectures, the lecturer will explain the theoretical foundations of the algorithms, discuss their assumptions and limitations, and illustrate their application to selected problems. Practical sessions will guide students through the implementation, testing and evaluation of reinforcement learning agents using Python and Jupyter Notebook.

Students will be encouraged to compare alternative solutions, interpret experimental results and discuss implementation difficulties. They will also practise documenting their work and explaining the reasoning behind their choices.

Students with valid certifications for Specific Learning Disabilities (SLD), disabilities, or other educational needs are invited to contact the teacher and the School Disability Coordinator at the beginning of the teaching unit to discuss possible teaching methods that, while respecting the teaching unit objectives, take into account individual learning styles. The contact details of the School Disability Coordinator are available at the following link: University Committee for the Inclusion of Students with Disabilities or SLD | UniGe | University of Genoa

SYLLABUS/CONTENT

The course covers the following topics:

• Agent architectures: organisation of autonomous agents and interaction with the environment.
• The reinforcement learning problem: states, actions, rewards, returns, policies and value functions.
• Dynamic programming: solution methods based on a known model of the environment.
• Exploration and exploitation: strategies for balancing information gathering and reward maximisation.
• Policy evaluation: estimation of the expected return associated with a policy.
• Policy improvement: methods for improving agent behaviour using value estimates.
• Model-based methods: use of environment models for planning and learning.
• Neural Fitted Q Iteration (NFQ): value function approximation using neural networks.
• Deep Q-Networks (DQN): value-based deep reinforcement learning.
• Policy-based and actor–critic methods: direct policy optimisation and combined policy and value learning.
• Deep Deterministic Policy Gradient (DDPG): learning policies for continuous action spaces.
• Proximal Policy Optimization (PPO): policy optimisation using constrained updates.
• Decision Transformers: an introduction to sequence modelling approaches to decision-making using recorded experience.

Practical activities focus on implementing algorithms, conducting experiments and comparing agent performance on selected tasks. Some teaching materials, including slides and source code for the practical exercises, are available in the course repository.

RECOMMENDED READING/BIBLIOGRAPHY

Slides, source code and other teaching materials will be available on aul@web. The materials provided, supplemented by notes taken during lectures and completion of the practical exercises, are sufficient to prepare for the examination.

The following books are recommended for further reading:

• Miguel Morales, Grokking Deep Reinforcement Learning, Manning.
• Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction, MIT Press.

Other university-level reinforcement learning textbooks may also be used as supplementary resources.

TEACHERS AND EXAM BOARD

LESSONS

LESSONS START

https://corsi.unige.it/corsi/11994/students-timetable

Class schedule

The timetable for this course is available here: Portale EasyAcademy

EXAMS

EXAM DESCRIPTION

The examination consists of a written test covering the theoretical and practical topics addressed during the course. It includes open-ended questions and problems concerning the formulation of reinforcement learning tasks, the operation and comparison of algorithms, their implementation in Python and the interpretation of experimental results.

Students must demonstrate both an understanding of the theoretical foundations and the ability to apply them to the development and evaluation of autonomous agents.

Further details on the examination format and the required depth of understanding will be provided during lectures.

Students with valid certifications for Specific Learning Disabilities (SLD), disabilities, or other educational needs may request compensatory and/or exemption measures for the exam. The procedures will be defined on a case-by-case basis, in accordance with the University Regulations, in consultation with the instructor and the School's Disability and SLD Coordinator.

ASSESSMENT METHODS

The written examination assesses students’ knowledge of reinforcement learning and their ability to apply it to the design, implementation and evaluation of autonomous agents.

Students must be able to:

• explain the concepts and algorithms covered, including their assumptions, conditions of application, advantages and limitations;
• formulate a problem in reinforcement learning terms and identify suitable solution methods;
• describe the main steps of an algorithm and translate them into Python code;
• justify methodological and implementation choices by comparing possible alternatives;
• interpret experimental results and identify potential causes of poor agent performance;
• present their reasoning clearly, using appropriate terminology and mathematical notation.

Assessment will consider the accuracy and completeness of answers, the correctness of the proposed solutions, the ability to connect theory and practice, independent judgement and clarity of technical communication.

FURTHER INFORMATION

For further information not included in the course description, please contact the lecturer.