OVERVIEW

This course introduces some of the main design issues for distributed systems (i.e., systems made of networked computers that communicate and coordinate by passing messages to each other).

Topics like performance, availability and security are discussed, and students make hands-on experience on simulated and real systems.

The course is presented in two versions: a 6 CFU version offered to the students of the Software Security & Engineering (SSE) curriculum, and a 9 CFU version offered to those of the Data Science & Engineering (DSE) curriculum. The 9 CFU version offers a more in-depth introduction to data-intensive systems such as Apache Hadoop and Spark

AIMS AND CONTENT

LEARNING OUTCOMES

Learning how to design high quality distributed systems, ranging from classical client-server to peer-to-peer and blockchain-based systems, and applying classical mathematical tools to measure reliability, availability, and fault tolerance.

AIMS AND LEARNING OUTCOMES

• EXPLAIN the reasons for using (or not using) distributed systems
• DISCUSS the key motivations and ideas for solutions of the problem of distributed consensus
• SUMMARIZE some important results about scheduling for distributed systems
• DISCUSS the main motivations, issues and solutions related to the design of peer-to-peer architectures
• EXPLAIN the design of large-scale data processing tools
• EVALUATE the impact of some design choices through simulation

For the 9 CFU version of the course,

• SOLVE programming tasks efficiently using large-scale data processing tools

PREREQUISITES

To be successful in this course, students should already know:

• data structures and algorithms
• client-server architectures
• network protocols
• basic security topics

Programming skills are required for the practical activities, which will be carried out in Python.

TEACHING METHODS

Lectures, practical exercises, and individual study.

SYLLABUS/CONTENT

• Transactions and Consensus
  • ACID properties and the CAP theorem
  • Consensus algorithms: Paxos, Raft
  • Concrete applications: Zookeeper, Google Cloud Spanner
• Queueing and Scheduling
  • Some theory on the problem of prioritizing and dispatching jobs to workers in a distributed system
• Erasure Coding
  • How to efficiently guarantee data availability
• Peer-to-Peer (P2P) Systems
  • Tor: a system for enabling anonymous communication
  • Searching in P2P systems: unstructured networks, superpeers, distributed hash tables
  • Incentives to cooperation: BitTorrent
  • Eventual consistency: the example of Amazon Dynamo
• Introduction to Large-Scale Data Processing
  • The MapReduce paradigm
  • The design of Hapace Hadoop and Apache Spark

For the 9 CFU version:

• Distributed data systems: overview, partitioning, replication
• Containerized architecture: Docker
• Practices with PySpark

RECOMMENDED READING/BIBLIOGRAPHY

Scientific papers covering the topics of the syllabus will be suggested during the course. The course material will be presented on Aulaweb.

TEACHERS AND EXAM BOARD

Exam Board

GIORGIO DELZANNO (President)

MARINA RIBAUDO

MATTEO DELL'AMICO (President Substitute)

LESSONS

LESSONS START

In agreement with the calendar approved by the Degree Program Board of Computer Science.

EXAMS

EXAM DESCRIPTION

6 CFU version:

Oral examination with discussion of the practical exercises assigned during the course and theory introduced during lectures

9 CFU version:

Evaluation of assignments and written exam.

ASSESSMENT METHODS

For each assignment, the quality of the code and the reports will indicate the technical skills acquired by the students.

The oral (6 CFU) and written (9 CFU) exam will allow the teachers to complete their evaluation with questions on the course program. 

FURTHER INFORMATION

See the AulaWeb page for the course.