Getting Access to LUMI

Who pays the bills?

LUMI is one of the larger EuroHPC supercomputers. EuroHPC currently funds supercomputers in three different classes:

1. There are a number of so-called petascale supercomputers. The first ones of those are Meluxina (in Luxembourg), VEGA (in Slovenia), Karolina (in the Czech Republic), Discoverer (in Bulgaria) and Deucalion (in Portugal).

2. A number of pre-exascale supercomputers, LUMI being one of them. The other two are Leonardo (in Italy) and MareNostrum 5 (in Spain)

3. A decision has already been taken on two exascale supercomputers: Jupiter (in Germany) and Jules Verne (consortium name) (in France).

Depending on the machine, EuroHPC pays one third up to half of the bill, while the remainder of the budget comes from the hosting country, usually with the help of a consortium of countries. For LUMI, EuroHPC paid half of the bill and is the actual owner of the machine.

LUMI is hosted in Finland but operated by a consortium of 11 countries. Each LUMI consortium country can set its own policies for a national access program, within the limits of what the supercomputer can technically sustain. There is a "Get Started" page on the main LUMI website with links to the various access programs.

Web links:

• EuroHPC JU supercomputers
• LUMI website "Get Started" page with links to relevant web pages for EuroHPC access and access via consortium countries.

Users and projects

LUMI works like most European large supercomputers: Users are members of projects.

A project corresponds to a coherent amount of work that is being done by a single person or a collaboration of a group of people. It typically corresponds to a research project, though there are other project types also, e.g., to give people access in the context of a course, or for organisational issues, e.g., a project for a local support organisation. Most projects are short-lived, with a typical duration of 4 to 6 months for benchmarking projects or one year for a regular project (which is the maximum duration agreed upon in the consortium).

Projects are also the basis for most research allocations on LUMI. In LUMI there are three types of resource allocations, and each project needs at least two of them:

1. A compute budget for the CPU nodes of LUMI (LUMI-C and the CPU-only large memory nodes), expressed in core-hours.
2. A compute budget for the GPU nodes of LUMI (LUMI-G and the visualisation nodes), expressed in GPU-hours. As the mechanism was already fixed before it became publicly known that for all practical purposes one AMD MI250X GPU should really be treated as 2 GPUs, one GPU-hour is one hour on a full MI250X, so computing for one hour on a full LUMI-G GPU node costs 4 GPU-hours.
3. A storage budget which is expressed in TB-hours. Only storage that is actually being used is charged on LUMI, to encourage users to clean up temporary storage. The rate at which storage is charged depends on the file system, but more about that later when we discuss the available file spaces.

These budgets are assigned and managed by the resource allocators, not by the LUMI User Support Team. The LUMI User Support Team also cannot add additional billing units to your project.

LUMI projects will typically have multiple project numbers which may be a bit confusing:

1. Each RA may have its own numbering system, often based on the numbering used for the project requests. Note that the LUMI User Support Team is not aware of that numbering as it is purely internal to the RA.

2. Each project on LUMI also gets a LUMI project ID which also corresponds to a Linux group to manage access to the project resources. These project IDs are of the form project_465XXXXXX for most projects but project_462XXXXXX for projects that are managed by the internal system of CSC Finland.

   This is also the project number that you should mention when contacting the central LUMI User Support.

Besides projects there are also user accounts. Each user account on LUMI corresponds to a physical person, and user accounts should not be shared. Some physical persons have more than one user account but this is an unfortunate consequence of decisions made very early in the LUMI project about how projects on LUMI would be managed. Users themselves cannot do a lot without a project as all a user has on LUMI is a small personal disk space which is simply a Linux requirement. To do anything useful on LUMI users need to be member of a project. There are also "robot accounts" for special purposes that would not correspond to a physical person but have a specific goal (like organising data ingestion from an external source) and few projects are granted such an account.

There ia a many-to-many mapping between projects and user accounts. Projects can of course have multiple users who collaborate in the project, but a user account can also be part of multiple projects. The latter is more common than you may think, as. e.g., you may become member of a training project when you take a LUMI training.

Most resources are attached to projects. The one resource that is attached to a user account is a small home directory to store user-specific configuration files. That home directory is not billed but can also not be extended. For some purposes you may have to store things that would usually automatically be placed in the home directory in a separate directory, e.g., in the project scratch space, and link to it. This may be the case when you try to convert big docker containers into singularity containers as the singularity cache can eat a lot of disk space. (Or sometimes setting an environment variable is enough to redirect to a different directory.)

Project management

A large system like LUMI with many entities giving independent access to the system to users needs an automated system to manage those projects and users. There are two such systems for LUMI. CSC, the hosting institution from Finland, uses its own internal system to manage projects allocated on the Finnish national share. This system manages the "642"-projects. The other system is called Puhuri and is developed in a collaboration between the Nordic countries to manage more than just LUMI projects. It can be used to manage multiple supercomputers but also to manage access to other resources such as experimental equipment. Puhuri projects can span multiple resources (e.g., multiple supercomputers so that you can create a workflow involving Tier-2, Tier-1 and Tier-0 resources).

There are multiple frontends in use for Puhuri. Some countries use their own frontend that links to the Puhuri backend to give their users a familiar feeling, while other countries use a Puhuri frontend that they either host and manage themselves, or run on the Puhuri infrastructure. Due to this diversity, we cannot really demo project management in the course but need to leave this to the local organisations.

The login to Puhuri is in general via MyAccessID, which is a GÉANT service. GÉANT is the international organisation that manages the research network in Europe. MyAccessID then in turn connects to your institute identity provider and a number of alternatives. It is important that you always use the same credentials to log in via MyAccessID, otherwise you create another user in MyAccessID that is unknown to Puhuri and get all kinds of strange error messages. MyAccessID is also used for ssh key management, so that in the future, when MyAccessID might serve more machines, you'd have a single set of ssh keys for all infrastructures.

Puhuri can be used to check your remaining project resources, but once your user account on LUMI is created, it is very easy to do this on the command line with the lumi-workspaces command.

Web links

• Puhuri documentation, look for the "User Guides".

• The lumi-workspaces command is provided through the lumi-tools module which is loaded by default. The command will usually give the output you need when used without any argument.

File spaces

LUMI has file spaces that are linked to a user account and file spaces that are linked to projects.

Per-user file spaces

The only permanent file space linked to a user account is the home directory which is of the form /users/<my_uid>. It is limited in both size and number of files it can contain, and neither limit can be expanded. It should only be used for things that are not project-related and first and foremost for those things that Linux and software automatically stores in a home directory like user-specific software configuration files. It is not billed as users can exist temporarily without an active project but therefore is also very limited in size.

Per-project file spaces

Each project also has 4 permanent or semi-permanent file spaces that are all billed against the storage budget of the project.

1. Permanent (for the duration of the project) storage on a hard disk based Lustre filesystem accessed via /project/project_46YXXXXXX. This is the place to perform the software installation for the project (as it is assumed that a project is a coherent amount of work it is only natural to assume that everybody in the project needs the same software), or to store input data etc. that will be needed for the duration of the project.

   Storing one TB for one hour on the disk based Lustre file systems costs 1 TB-hour. As would storing 10 GB for 100 hours.

2. Semi-permanent scratch storage on a hard disk based Lustre filesystem accessed via /scratch/project_46YXXXXXX. Files in this storage space can in principle be erased automatically after 90 days. This is not happening yet on LUMI, but will be activated if the storage space starts to fill up.

   Storing one TB for one hour on the disk based Lustre file systems costs 1 TB-hour.

3. Semi-permanent scratch storage on an SSD based Lustre filesystem accessed via /flash/project_46YXXXXXX. Files in this storage space can in principle be erased automatically after 30 days. This is not happening yet on LUMI, but will be activated if the scratch storage space starts to fill up.

   Storing one TB for one hour on the flash based Lustre file system costs 10 TB-hour, also reflecting the purchase cost difference of the systems.

4. Permanent (for the duration of the project) storage on the hard disk based object filesystem.

   Storing one TB for one hour on the object based file system costs 0.5 TB-hour.

Quota

The slide above also shows the quota on each volume. This information is also available in the LUMI docs.

The use of space in each file space is limited by block and file quota. Block quota limit the capacity you can use, while file quota limit the number of so-called inodes you can use. Each file, each subdirectory and each link use an inode. As we shall see later in this course (in the section on Lustre) or as you may have seen in other HPC courses already, parallel file systems are not built to deal with hundreds of thousands of small files and are very inefficient at that. Therefore block quota on LUMI tend to be rather flexible (except for the home directory) but file quota are rather strict and will not easily get extended. Software installations that require tens of thousands of small files should be done in containers (e.g., conda installations or any big Python installation) while data should also be organised in proper file formats rather than being dumped on the file system abusing the file system as a database.

In the above slide, the "Capacity" column shows the block quota and the "Files" column show the total number of so-called inodes available in the file space.

The project file spaces can be expanded in capacity within the limits specified. However, as big parallel file systems are very bad at handling lots of small files (see also the session on Lustre), the files quota (or more accurately inode quota) are rather strict and not easily raised (and if raised, not by an order of magnitude).

So storage use on LUMI is limited in two independent ways:

• Traditional Linux block and file quota limit the maximum capacity you can use (in volume and number of inodes, roughly the number of files and directories combined).

• But actual storage use is also "billed" on a use-per-hour basis. The idea behind this is that a user may run a program that generates a lot of data, but after some post-processing much of the data can be deleted so that other users can use that capacity again, and to encourage that behaviour you are billed based not on peak use, but based on the combination of the volume that you use and the time you use it for.

  Storage use is monitored hourly for the billing process. If you run out of storage billing units, you will not be able to run jobs anymore.

  Storage in your home directory is not billed but that should not mean that you should abuse your home directory for other purposes then a home directory is meant to be used, and an extension of the home directory will never be granted. If you run out of space for, e.g., caches, you should relocate them to, e.g., your scratch space, which can sometimes be done by setting an environment variable and in other cases by just using symbolic links to preserve the structure of the caching subdirectories in your home directory while storing data elsewhere.

Quota extensions are currently handled by the central LUMI User Support Team. But storage billing units, just as any billing unit, comes from your resource allocator, and the LUMI User Support Team cannot give you any storage billing units.

Some additional information

LUMI has four disk based Lustre file systems that house /users, /project and /scratch. The /project and /scratch directories of your project will always be on the same parallel file system, but your home directory may be on a different one. Both are assigned automatically during project and account creation and these assignments cannot be changed by the LUMI User Support Team. As there is a many-to-many mapping between user accounts and projects it is not possible to ensure that user accounts are on the same file system as their main project. In fact, many users enter LUMI for the first time through a course project and not through one of their main compute projects...

It is important to note that even though /flash is SSD based storage, it is still a parallel file system and will not behave the way an SSD in your PC does. The cost of opening and closing a file is still very high due to it being both a networked and a parallel file system rather than a local drive. In fact, the cost for metadata operations is not always that much lower as on the hard disk based parallel file systems as both use SSDs to store the metadata (but some metadata operations on Lustre involve both the metadata and object servers and the latter are faster on /flash). Once a file is opened and with a proper data access pattern (big accesses, properly striped files which we will discuss later in this course) the flash file system can give a lot more bandwidth than the disk based ones.

It is important to note that LUMI is not a data archiving service or a data publishing service. "Permanent" in the above discussion only means "for the duration of the project". There is no backup, not even of the home directory. And 90 days after the end of the project all data from the project is irrevocably deleted from the system. User accounts without project will also be closed, as will user accounts that remain inactive for several months, even if an active project is still attached to them.

If you run out of storage billing units, access to the job queues or even to the storage can be blocked and you should contact your resource allocator for extra billing units. It is important that you clean up after a run as LUMI is not meant for long-term data archiving. But at the same time it is completely normal that you cannot do so right after a run, or as a job may not launch immediately, that you need to put input data on the system long before a run starts. So data needed for or resulting from a run has to stay on the system for a few days or weeks, and you need to budget for that in your project request.

Web links:

• Overview of storage systems on LUMI
• Billing policies (includes those for storage)

Access

LUMI currently has 4 login nodes through which users can enter the system via key-based ssh. The generic name of those login nodes is lumi.csc.fi. Using the generic names will put you onto one of the available nodes more or less at random and will avoid contacting a login node that is down for maintenance. However, in some cases one needs to enter a specific login node. E.g., tools for remote editing or remote file synchronisation such as Visual Studio Code or Eclipse usually don't like it if they get a different node every time they try to connect, e.g., because they may start a remote server and try to create multiple connections to that server. In that case you have to use a specific login node, which you can do through the names lumi-uan01.csc.fi up to lumi-uan04.csc.fi. (UAN is the abbreviation for User Access Node, the term Cray uses for login nodes.)

Key management is for most users done via MyAccessID: mms.myaccessid.org. This is the case for all user accounts who got their first project on LUMI via Puhuri. User accounts that were created via the My CSC service have to use the my.csc.fi portal to manage their keys. It recently became possible to link your account in My CSC to MyAccessID so that you do not get a second account on LUMI ones you join a Puhuri-managed project, and in this case your keys are still managed through the My CSC service.

LUMI now also provides a web interface via Open OnDemand. The URL is https://www.lumi.csc.fi/. It also offers a number of tools that can be useful for visualisation via a web browser, but it is still work-in-progress.

There is currently moderate support for technologies for GUI applications on LUMI. Running X11 over ssh (via ssh -X) is unbearably slow for most users as X11 is not meant to be used over long-distance networks and is very latency-sensitive. The alternative is VNC, which we currently offer in two different ways:

• Via the "Desktop" app in Open OnDemand, which will give you a VNC session with the rather lightweight Xfce desktop environment,
• and through the lumi-vnc module which was our primary method when Open OnDemand for LUMI was not ready yet.

You can connect through a web browser or a VNC client. Don't expect more advanced desktop environments: LUMI is not meant to be your remote graphics workstation and we cannot afford to spend tens of compute nodes on offering this service.

Web links:

• LUMI documentation on logging in to LUMI and creating suitable SSH keys
• CSC documentation on linking My CSC to MyAccessID

A walk through the Open OnDemand interface

To enter the LUMI OpenOndemand interface, point your browser to www.lumi.csc.fi. You will get the screen:

Most likely you just want to log on, so click on "Go to login" and the "Select authentication provider" screen should appear, with a link to give you more information about which authentication method to use:

Basically, if you are a CSC user (and definitely when you're on a CSC 462* project) you'll want to use the "CSC" or "Haka" choice, while other users will need MyAccessID.

The whole login process is not shown, but after successful authentication, you end up at the main screen (that you can also go back to by clicking the LUMI-logo in the upper left corner):

The list of pinned apps may change over time, and more apps are available via the menu at the top. Most apps will run in the context of a job, so you will need billing units, and those apps will also present you with a form to chose the resources you want to use, but that will only be discussed in the session on Slurm.

Two apps don't run in the context of a job: The "Login node shell" and "Home Directory" apps, and we'll first have a look at those.

The "Login node shell" does just what you expect from it": It opens a tab in the browser with a shell on one of the login nodes. Open OnDemand uses its own set of login nodes, as you can see from the name of the node, but these nodes are otherwise identical to the login nodes that you access via an ssh client on your laptop, and the same policies apply. They should not be used for running applications and only be used for small work or not too heavy compilations.

Let's now select the "Home Directory" app. We get:

The "Home Directory" app presents you with an interface through which you cannot only browse your home directory, but also the project, scratch and flash directories of all your projects. It can be used for some elementary file access and also to upload and download files.

It is not suitable though to upload or download very big files, or download large subdirectories (multiple files will be packed in a ZIP archive) as browsers may not be reliable enough and as there are also restrictions on how big an archive Open OnDemand can create.

For transferring lots of data, transfer via LUMI-O is certainly the better option at the moment.

Finally, let's have a look at the "Desktop" app.

The "Desktop" app will present you with a simple GUI desktop based on the Xfce desktop environment. This app needs to run in the context of a job and although it can run on several partitions on LUMI, its main use is to be able to use some visualisation applications, so your best choice is likely to use the partition with visualisation GPUs (see the session on Slurm). As we have not discussed jobs yet, we will skip how to fill in the form that is presented to you.

The desktop is basically run in a VNC session, a popular protocol for remote desktop support in Linux. It can be used through a web browser, which is what you get if you click the "Launch Desktop" button, but there are other choices also.

After launching/connecting to the desktop you get:

There is a small settings menu hidden at the left (expanded in the picture) to do some settings of the web interface that we are using here. Right-clicking with the mouse on the desktop gives you a menu with a number of applications.

This is in no means meant to be a replacement of your own workstation, so the software choice is limited and will remain limited. It should never be your main environment for all your work. LUMI is not meant to simply provide small workstations to all of Europe. And it will also react a lot slower than what you are used to from a workstation in front of you. This is 100% normal and simply the result of using a computer which is far away so there is a high network latency.

Data transfer

There are currently two main options to transfer data to and from LUMI.

The first one is to use sftp to the login nodes, authenticating via your ssh key. There is a lot of software available for all major operating systems, both command line based and GUI based. The sftp protocol can be very slow over high latency connections. This is because it is a protocol that opens only a single stream for communication with the remote host, and the bandwidth one can reach via a single stream in the TCP network protocol used for such connections, is limited not only by the bandwidth of all links involved but also by the latency. After sending a certain amount of data, the sender will wait for a confirmation that the data has arrived, and if the latency is high, that confirmation takes more time to reach the sender, limiting the effective bandwidth that can be reached over the connection. LUMI is not to blame for that; the whole path from the system from which you initiate the connection to LUMI is responsible and every step adds to the latency. We've seen many cases where the biggest contributor to the latency was actually the campus network of the user.

The second important option is to transfer data via the object storage system LUMI-O. To transfer data to LUMI, you'd first push the data to LUMI-O and then on LUMI pull it from LUMI-O. When transferring data to your home institute, you'd first push it onto LUMI-O from LUMI and then pull the data from LUMI-O to your work machine. LUMI offers some support for various tools, including rclone and S3cmd. There also exist many GUI clients to access object storage. Even though in principle any tool that can connect via the S3 protocol can work, the LUMI User Support Team cannot give you instructions for every possible tool. Those tools for accessing object storage tend to set up multiple data streams and hence will offer a much higher effective bandwidth, even on high latency connections.

Alternatively, you can also chose to access external servers from LUMI if you have client software that runs on LUMI (or if that software is already installed on LUMI, e.g., rclone and S3cmd), but the LUMI User Support Team cannot tell you how to configure tools to use an external service that we have no access to.

Unfortunately there is no support yet for Globus or other forms of gridFTP.

What is LUMI-O?

LUMI-O is an object storage system (based on Ceph). Users from Finland may be familiar with Allas, which is similar to the LUMI object storage system, though LUMI doesn't provide all the functionality of Allas.

Object file systems need specific tools to access data. They are usually not mounted as a regular filesystem (though some tools can make them appear as a regular file system) and accessing them needs authentication via temporary keys that are different from your ssh keys and are not only bound to you, but also to the project for which you want to access LUMI-O. So if you want to use LUMI-O for multiple projects simultaneously, you'll need keys for each project.

Object storage is not organised in files and directories. A much flatter structure is used with buckets that contain objects:

• Buckets: Containers used to store one or more objects. Object storage uses a flat structure with only one level which means that buckets cannot contain other buckets.

• Objects: Any type of data. An object is stored in a bucket.

• Metadata: Both buckets and objects have metadata specific to them. The metadata of a bucket specifies, e.g., the access rights to the bucket. While traditional file systems have fixed metadata (filename, creation date, type, etc.), an object storage allows you to add custom metadata.

Objects can be served on the web also. This is in fact how recordings of some of the LUST courses are served currently. However, LUMI-O is not meant to be used as a data publishing service and is not an alternative to services provided by, e.g., EUDAT or several local academic service providers.

The object storage can be easily reached from outside LUMI also. In fact, during downtimes, LUMI-O is often still operational as its software stack is managed completely independently from LUMI. It is therefore also very well suited as a mechanism for data transfer to and from LUMI. Moreover, tools for object storage often perform much better on high latency long-distance connections than tools as sftp.

LUMI-O is based on the Ceph object file system. It has a total capacity of 30 PB. Storage is persistent for the duration of a project. Projects get a quota of 150 TB and can create up to 1K buckets and 500K objects per bucket. These quota are currently fixed and cannot be modified. Storage on LUMI-O is billed at 0.5 TB·hour per TB per hour, half that of /scratch or /project. It can be a good alternative to store data from your project that still needs to be transferred but is not immediately needed by jobs, or to maintain backups on LUMI yourself.

Accessing LUMI-O

Access to LUMI-O is based on temporary keys that need to be generated via a web interface (though there may be alternatives in the future).

There are currently three command-line tools pre-installed on LUMI: rclone (which is the easiest tool if you want public and private data), s3cmd and restic.

But you can also access LUMI-O with similar tools from outside LUMI. Configuring them may be a bit tricky and the LUMI User Support Team cannot help you with each and every client tool on your personal machine. However, the web interface that is used to generate the keys, can also generate code snippets or configuration file snippets for various tools, and that will make configuring them a lot easier.

In the future access via Open OnDemand should also become possible.

Key generation

Keys are generated via a web interface that can be found at auth.lumidata.eu. In the future it should become possible to do so directly in the Open OnDemand interface, and may even from the command line.

Let's walk through the interface:

A walk through the credentials management web interface of LUMI-O

After entering the URL auth.lumidata.eu, you're presented with a welcome screen on which you have to click the "Go to login" button.

This will present you with the already familiar (from Open OnDemand) screen to select your authentication provider:

Proceed with login in through your relevant authentication provider (not shown here) and you will be presented with a screen that show your active projects:

Click the project for which you want to generate a key, and the column to the right will appear. Chose how long the key should be valid (1 week or 168 hours is the maximum currently, but the life can be extended) and a description for the key. The latter is useful if you generate multiple keys for different use. E.g., for security reasons you may want to use different keys from different machines so that one machine can be disabled quickly if the machine would be compromised or stolen.

Next click on the "Generate key" button, and a new key will appear in the "Available keys" section:

Now click on the key to get more information about the key:

At the top of the screen you see three elements that will be important if you use the LUMI command line tool lumio-conf to generate configuration files for rclone and s3cmd: the project number (but you knew that one), the "Access key" and "Secret key".

Scrolling down a bit more:

The "Extend key" field can be used to extend the life of the key, to a maximum of 168 hours past the current time.

The "Configuration templates" is the way to generate code snippets or configuration file snippets for various tools (see the list on the slide). After selecting "rclone" and clicking the "Generate" button, a new screen opens:

This screen shows us the snippet for the rclone configuration file (on Linux it is ~/.config/rclone/rclone.conf). Notice that it creates to so-called endpoints. In the slide this is lumi-465001102-private and lumi-465001102-public, for storing buckets and objects which are private or public (i.e., also web-accessible).

Configuring LUMI-O tools

On LUMI, you can use the lumnio-conf tool to configure rclone and s3cmd. To access the tool, you need to load the lumio module first, which is always available. The same module will also load a module that makes rclone, s3cmd and restic available.

Whe starting lumio-conf, it will present with a couple of questions: The project number associated with the key, the access key and the secret key. We have shown above where in the web interface that information can be found. A future version may or may not be more automatic. As we shall see in the next slide, currently the rclone configuration generated by this tool is (unfortunately) different from the one generated by the web interface.

Another way to configure tools for object storage access is simply via the code snippets and configuration files snippets as has already been discussed before. The same snippets should also work when you run the tools on a different computer.

rclone on LUMI-O

The rclone configuration file for LUMI-O contains two end points, and unfortunately at the moment both ways discussed on the previous slide, produce different end points.

• When using lumio-conf, you'll get:
  • lumi-o as the end point for buckets and object that should be private, i.e., not publicly accessible via the web interface, and
  • lumi-pub for buckets and objects that should be publicly accessible. It does appear to be possible to have both types in a single bucket though.
• When using the web generator you get specific end points for each project, so it is possible to access data from multiple projects simultaneously from a single configuration file:
  • lumi-46YXXXXXX-private is the end point to be used for buckets and objects that should be private, and
  • lumi-46YXXXXXX-public is the end point for data that should be publicly accessible.

A description of the main rclone commands is outside the scope of this tutorial, but some options are discussed in the LUMI documentation, and the same page also contains some documentation for s3cmd and restic. See the links below for even more documentation.

Further LUMI-O documentation

• Documentation for the LUMI-O object storage service

• Software for LUMI-O on LUMI is provided through the lumio module which provides the configuration tool on top of the software and the lumio-ext-tools module providing rclone, S3cmd and restic and links to the documentation of those tools.

  • rclone documentation
  • S3cmd tools usage
  • restic documentation