OVERVIEW

The aim of this course is to provide basic knowledge on the architecture of parallel processing systems, along with a minimum of programming skills essential to using such systems. As far as the programming part is concerned, the emphasis is on parallel message-passing programming using MPI, but the OpenMP shared memory paradigm is also practiced.  GPU architecture is presented, and GPU programming with OpenCL is practiced.  Architectural aspects of large-scale computing platforms are presented.

AIMS AND CONTENT

LEARNING OUTCOMES

Students will be provided with knowledge in the basic aspects of modern high-performance computing systems (pipeline/superscalar processors, shared-memory/message-passing multiprocessors, vector processors, GPUs, cloud-based platforms) and basic programming skills for high-performance computing (cache optimization, OpenMP, MPI, OpenCL, map-reduce/dataflow). Students will be involved in project activities.

TEACHING METHODS

Lessons, practicals, individual study with textbooks

SYLLABUS/CONTENT

1. Performance of a computer:
   Direct and inverse, absolute and relative performance indices. Benchmarks.
2. Pipeline processor architecture:
   Performance of a pipeline system and its analytical evaluation. Overall structure of a pipeline processor. Pipeline hazards (structural, data, control) and their impact on performance. Reducing hazards and/or their impact: hardware techniques. Instruction-level parallelism in sequential programs. How to make better use of instruction-level parallelism when using pipeline processors: loop unrolling, instruction reordering.
3. Advanced pipeline processors:
   Dynamic Instruction Scheduling with Tomasulo Architecture.
   Branch Prediction.
   Speculative execution of instructions.
4. Superscalar processors:
   Multiple-issue processors. Scheduling instructions on multiple-issue processors. VLIW processors.
5. Cache Memory and Computer Performance:
   Hardware techniques to reduce cache miss penalties. Hardware and software techniques to reduce cache miss frequency. Hardware techniques to hide cache miss overheads. Practicals with matrix multiplication.
6. Multiprocessor computers:
   The purpose of a parallel computer. Limits of parallel computers: Amdahl's law, communication delays. MIMD computers: shared memory, distributed memory with shared address space, distributed memory with message-passing.
7. MIMD shared-memory computers:
   Overall organization. Reducing memory access contention. Cache coherency: snooping-based protocols.
8.  
9. MIMD computers with distributed memory and shared address space:
   Overall organization. Directory-based cache coherence protocols.
10. Cooperation among processes on shared address space:
    Communication and synchronization. Synchronization algorithms: lock/unlock and barrier synchronization on shared address space and their performance.
11. Consistency in main memory. Weak consistency models and their performance benefits.
12. High-level parallel programming on shared address space: the OpenMP directives. Practicals with OpenMP.
13. Message-passing MIMD computers:
    Overall organization. Cooperation among processes: message-passing communication and synchronization. Blocking vs. non-blocking, point-to-point vs. collectives, implementation aspects. Non-blocking communication and instruction reordering.
14. High-level parallel programming with message-passing:
    the SPMD paradigm, the Message Passing Interface (MPI) standard. Practicals with MPI.
15. The concept of load balancing and its impact on performance.
    Dynamic load balancing: the "farm" parallel structure and its performance analysis.
16. SIMD parallel computers: vector processors. Modern descendants of SIMD computers: vector extensions, GPUs.
    GPU programming with OpenCL. Practicals with OpenCL on a GPU.
17. Parallel I/O and MPI-I/O.
18. Architecture of large-scale computing platforms.

RECOMMENDED READING/BIBLIOGRAPHY

John Hennessy, David Patterson: Computer architecture: a quantitative approach (fifth edition), Morgan Kaufmann.

Here is the online version of Jan Foster's book: Designing and Building Parallel Programs, Addison Wesley.

Slides, tutorials, and code samples, that can be found on Aulaweb, integrate but do not replace the textbooks.

TEACHERS AND EXAM BOARD

Exam Board

ANNALISA BARLA (President)

GIUSEPPE CIACCIO (President)

GIORGIO DELZANNO (President)

LESSONS

Class schedule

The timetable for this course is available here: Portale EasyAcademy

EXAMS

EXAM DESCRIPTION

The exam consists of an oral discussion on the topics of the course. The evaluation takes into account the work done during the practicals. Those who do not participate in the practicals shall take an additional practical test on parallel programming.

FURTHER INFORMATION

Prerequisites:  basic knowledge of computer architecture, fair programming skills.

More info at http://www.disi.unige.it/person/CiaccioG/hpc.html