LUMI Intensive: Comprehensive Intro and Advanced Workshop, March 3-7, 2025¶

Target audience¶

The course is an "on-site first course": We will broadcast the talks via Zoom, but there is direct interaction only between the lecturers and the people in the physical meeting room. Several members of LUST will be present for the whole week, and specialists from HPE and AMD will join on day 3 till 5, so being on-site gives you excellent opportunities to work with these people in a way that is impossible via Zoom or email. Users taking the course on-line will be able to submit written questions. However, not all types of questions can be answered that way as some questions require direct interaction. Also, experience has shown that discussions that often unwind especially in the advanced part, are hard to follow on-line. They are not strictly moderated the way a discussion in a TV studio is, but are natural discussions.

Day 1 and 2 of the course are at a more introductory level. It covers all the essentials of LUMI: the system architecture, programming environments, software setup, running jobs efficiently, various types of data storage, and using containers which are essential for Python etc.

This part of the course is largely equivalent to the previous 2-day intro courses in May 2024 and December 2024.
Days 3-5 of the course are at a more advanced level and focus on the needs of developers of software for LUMI and users who want to better understand the behaviour of applications they are running on LUMI. There is some attention to AI also but users only interested in training models and not in gaining a deeper insight of performance analysis etc., may be better of with the AI trainings offered by LUST.

Part of the lectures are about tools that HPE and AMD offer to analyse program behaviour, and other lectures focus on a specific performance bottleneck, how to detect if it is present and how to deal with it.

The materials from day 1 and 2 are an absolute prerequisite to be able to benefit from this part of the course.

It is possible though to register only for the 3 advanced days if you have the required knowledge.

Together both parts essentially cover the materials from previous 4-day advanced courses (e.g., the course in Amsterdam in October 2024), but spending a bit more time on more elementary materials and with some new material on object storage and containers on LUMI.

See the draft schedule to get a better idea of topics that will be covered.

People registering for the whole event have priority for on-site seating on day 1 and 2.

Course organisation¶

Location: KTH, Lindstedtsvägen 24, 114 28Stockholm, Sweden

The building is just a 5 minute walk from a metro station with excellent connections to the central station (red metro line). See also the SL web site for more information on public transportation in Stockholm.
Schedule (some speakers will change)
HedgeDoc for questions
During the course, there are several Slurm reservations available:
- Day 2, for the Slurm exercises:
  - CPU nodes: LUMI_intro_cpu on the small partition
  - GPU nodes: LUMI_intro_gpu on the standard-g partition
- Day 3-5:
  - CPU nodes: LUMI_advanced_cpu on the standard partition
  - GPU nodes: LUMI_advanced_gpu on the standard-g partition
They can be used in conjunction with the training project project_465001726.

Note that the reservations and course project should only be used for making the exercises during the course and not for running your own jobs. The resources allocated to the course are very limited.

Travel¶

Hotel options: The various hotels near the central station would be a good choice for travellers. The Odenplan area could be an alternative as there is a frequent bus service to the KTH campus. Elite Hotel Arcadia is right on the edge of the KTH campus and hence also a possible choice.
Travel from the airport: There are 4 ways to get to central stockholm from the airport:
- Taxi - Expensive, go to the labeled taxi ranks in the airport. There is a fixed price to the city, which should be displayed on the taxi and will be different for different companies
- Arlanda Express - Moderately expensive, very fast direct train to stockholm central station
- Commuter train - From sky city in Arlanda you can get the nomral trains. You need to pay 147 SEK (toll to use the tunnel under Arlanda) at the ticket barriers plus a normal SL ticket
- Airport bus - cheapest option, but takes longest. Central bus station is right next to central station.
Using public transportation in Stockholm

Probably simplest if you just get the SL (Stockholm Localtrafik) app on a smartphone, will give you a QR code that you can use at ticket barriers. Tickets can be bought using a credit card in the app. Physical cards are also available, but cost 50 SEK for the card, plus extra for the tickets you put on it. Credit cards can also be used with contactless pay as you go to buy a single.

Course materials¶

Course materials include the Q&A of each session, slides when available and notes when available. These materials will become available as the course progresses.

Due to copyright issues some of the materials are only available to current LUMI users and have to be downloaded from LUMI.

Note: Some links in the table below are dead and will remain so until after the end of the course.

Presentation	slides	notes	recording
/	/	/	/

Presentation	slides	notes	recording
Day 1
Welcome and Introduction	S	/	V
Introduction to the course notes	/	N	/
*Theme: Exploring LUMI from the login nodes*
LUMI Architecture	S	(N)	V
HPE Cray Programming Environment	S	N	V
Getting Access to LUMI	S	N	V
Exercises 1	/	/	/
Modules on LUMI	S	N	V
Exercises 2	/	/	/
LUMI Software Stacks	S	N	V
Exercises 3	/	/	/
LUMI Support and Documentation	S	/	V
Wrap-Up Day 1	S	/	V
Day 2
Introduction Day 2	S	/	V
*Theme: Running jobs efficiently*
Slurm on LUMI	S	(N)	V
Process and Thread Distribution and Binding	S	(N)	V
Exercises 4	/	/	/
Theme: Data on LUMI
Using Lustre	S	(N)	V
Object Storage	S	N	V
Exercises 5	/	/	/
Theme: Containers on LUMI
Containers on LUMI-C and LUMI-G	S	N	V
Demo 1 (optional)	/	N	V
Demo 2 (optional)	/	N	V
Wrap-Up Day 2	S	/	V
Day 3
Introduction	/	/	web
HPE Cray EX Architecture	lumi	/	lumi
Compilers and Parallel Programming Models	lumi	/	lumi
Exercises #6	/	/	/
Cray Scientific Libraries	lumi	/	lumi
Exercises #7	/	/	/
CCE Offloading Models	lumi	/	lumi
Frameworks for porting applications to GPUs	lumi	/	lumi
Introduction to the AMD ROCm Ecosystem	web	/	web
Exercises #8	/	/	/
Debugging at Scale	lumi	/	lumi
Exercises #9	/	/	/
Day 4
Introduction to Perftools	lumi	/	lumi
Exercises #10	/	/	/
Performance Optimization: Improving single-core Efficiency	lumi	/	lumi
Advanced Performance Analysis	lumi	/	lumi
Exercises #11	/	/	/
MPI Topics on the HPE Cray EX Supercomputer	lumi	/	lumi
Exercises #12	/	/	/
AMD Debugger: ROCgdb	web	/	web
Exercises #13	/	/	/
Introduction to ROC-Profiler (rocprof)	web	/	web
Exercises #14	/	/	/
Day 5
Introduction to Python on Cray EX	lumi	/	lumi
Optimizing Large Scale I/O	lumi	/	lumi
Exercises #15	/	/	/
Introduction to OmniTrace	web	/	web
Exercises #16	/	/	/
Introduction to Omniperf	web	/	web
Exercises #17	/	/	/
Tools in Action - An Example with Pytorch	web	/	web
Appendices
Additional documentation	/	web	/

Setting up for the exercises¶

During the course, you can use the training project project_465001726 for the exercises. After the course, it is still possible to make almost all exercises in your own project. They need only very little resources, but you will need both CPU and GPU resources.

Exercises on day 1 and day 2¶

The "Object Storage" exercises in Exercise session 5 do require data in the training project, so these exercises should really be made while the training project is still active. There is an alternative set of exercises missing only one element from the during-the-course version that can be made in your own project. For the exercises from Exercise session 4 you can no longer use the reservation if they are not made during the second course day (the reservation expires at 5pm CET/ 6pm EET that day).

Create a directory in the scratch of your project, or if you want to keep the exercises around for a while, in a subdirectory of your project directory or in your home directory (though we don't recommend the latter). Then go into that directory.

E.g., in the scratch directory of your project:
```
mkdir -p /scratch/project_465001726/course-20250303-$USER/Exercises/Intro
cd /scratch/project_465001726/course-20250303-$USER/Exercises/Intro
```
where you have to replace project_465001726 using the number of your own project.

If you have no other project on LUMI, you can also use the scratch of the course project project_465001726. Do use a personal subdirectory as in the following commands:
```
mkdir -p /scratch/project_465001726/$USER/Exercises/Intro
cd /scratch/project_465001726/$USER/Exercises/Intro
```

Now download the exercises and un-tar:

wget https://462000265.lumidata.eu/2p3day-20250303/files/exercises-LUST-20250303.tar.bz2
tar -xf exercises-LUST-20250303.tar.bz2

Link to the tar-file with the exercises and the bzip2-compressed version.

You're all set to go!

Exercises HPE presentations¶

During the course, the exercises and instructions are available on LUMI in /project/project_465001726/Exercises/HPE.

Copy them to the directory where you make the exercises, e.g.,

cd /scratch/project_465001726/course-20250303-$USER/Exercises
cp -r /project/project_465001726/Exercises/HPE HPE

Some exercises are themselves in tar files that you can unpack with tar xf <file>.tar or tar xf <file>.tar.gz.
Reservations are setup for use during the training (on LUMI-C and LUMI-G)

Use the following flags in the SLURM commands:
- -A project_465001726 --reservation=LUMI_advanced_cpu
- -A project_465001726 --reservation=LUMI_advanced_gpu
To run the examples either use above options with sbatch/srun/salloc or you can also set SLURM environment variables, e.g.
```
export SLURM_ACCOUNT=project_465001726
export SLURM_RESERVATION=LUMI_advanced_gpu
```
(to be repeated for variables with prefix SLURM_, SBATCH_, SALLOC_)
For convenience, we provide a script to setup your environment (copy from /project/project_465001726/Exercises/HPE):
- source lumi_c.sh for exercises on LUMI-C
- source lumi_g.sh for exercises on LUMI-G
It will change the prompt accordingly, remember to run exit before you switch environment
And yet another alternative is to use modules that we provide that do the same as the scripts, but don't change the prompt and don't put you in a new shell, so instead of using exit, you'd unload the module.
- First add the modules to the module search path:
```
module use /appl/local/training/modules/2p3day-20250303
```
- Then load the appropriate module:
  - module load exercises/advanced-C for exercises on LUMI-C
  - module load exercises/advanced-G for exercises on LUMI-G
- And if you're finished or want to run outside the reservation, simply use
```
module unload exercises
```

Exercises AMD presentations¶

During the course, the files needed for the exercises are available on LUMI in /project/project_465001726/Exercises/HPE. The exercises themselves are in an online document linked to from the exercise pages in the course materials.

To copy the exercises to where you make the exercises, e.g.,

cd /scratch/project_465001726/course-20250303-$USER/Exercises
cp -r /project/project_465001726/Exercises/AMD AMD

Links to documentation¶

The links to all documentation mentioned during the talks is on a separate page.