LEARNING OUTCOMES

Students will be provided with knowledge in the basic aspects of modern high-performance computing systems (pipeline/superscalar processors, shared-memory/message-passing multiprocessors, vector processors, GPUs, cloud-based platforms) and basic programming skills for high-performance computing (cache optimization, OpenMP, MPI, OpenCL, map-reduce/dataflow). Students will be involved in project activities.

PREREQUISITES

Basic knowledge of computer architecture, fair programming skills.

TEACHING METHODS

Lessons, practicals, individual study with textbooks

SYLLABUS/CONTENT

1. Performance of a computer:
   Direct and inverse, absolute and relative performance indices. Benchmarks.
2. Pipeline processor architecture:
   Performance of a pipeline system and its analytical evaluation. Overall structure of a pipeline processor. Pipeline hazards (structural, data, control) and their impact on performance. Reducing hazards and/or their impact: hardware techniques. Instruction-level parallelism in sequential programs. How to make better use of instruction-level parallelism when using pipeline processors: loop unrolling, instruction reordering.
3. Advanced pipeline processors:
   Dynamic Instruction Scheduling with Tomasulo Architecture.
   Branch Prediction.
   Speculative execution of instructions.
4. Superscalar processors:
   Multiple-issue processors. Scheduling instructions on multiple-issue processors. VLIW processors.
5. Cache Memory and Computer Performance:
   Hardware techniques to reduce cache miss penalties. Hardware and software techniques to reduce cache miss frequency. Hardware techniques to hide cache miss overheads. Practicals with matrix multiplication.
6. Multiprocessor computers:
   The purpose of a parallel computer. Limits of parallel computers: Amdahl's law, communication delays. MIMD computers: shared memory, distributed memory with shared address space, distributed memory with message-passing.
7. MIMD shared-memory computers:
   Overall organization. Reducing memory access contention. Cache coherency: snooping-based protocols.
8.  
9. MIMD computers with distributed memory and shared address space:
   Overall organization. Directory-based cache coherence protocols.
10. Cooperation among processes on shared address space:
    Communication and synchronization. Synchronization algorithms: lock/unlock and barrier synchronization on shared address space and their performance.
11. Consistency in main memory. Weak consistency models and their performance benefits.
12. High-level parallel programming on shared address space: the OpenMP directives. Practicals with OpenMP.
13. Message-passing MIMD computers:
    Overall organization. Cooperation among processes: message-passing communication and synchronization. Blocking vs. non-blocking, point-to-point vs. collectives, implementation aspects. Non-blocking communication and instruction reordering.
14. High-level parallel programming with message-passing:
    the SPMD paradigm, the Message Passing Interface (MPI) standard. Practicals with MPI.
15. The concept of load balancing and its impact on performance.
    Dynamic load balancing: the "farm" parallel structure and its performance analysis.
16. SIMD parallel computers: vector processors. Modern descendants of SIMD computers: vector extensions, GPUs.
    GPU programming with OpenCL. Practicals with OpenCL on a GPU.
17. Parallel I/O and MPI-I/O.
18. Architecture of large-scale computing platforms.

RECOMMENDED READING/BIBLIOGRAPHY

John Hennessy, David Patterson: Computer architecture: a quantitative approach (fifth edition), Morgan Kaufmann.

Here is the online version of Jan Foster's book: Designing and Building Parallel Programs, Addison Wesley.

Slides, tutorials, and code samples, that can be found on Aulaweb, integrate but do not replace the textbooks.