Explore ARCExplore ARC

ARC-TS begins work on new “Great Lakes” cluster to replace Flux

By | Flux, Happenings, HPC, News

Advanced Research Computing – Technology Services (ARC-TS) is starting the process of creating a new, campus-wide computing cluster, “Great Lakes,” that will serve the broad needs of researchers across the University. Over time, Great Lakes will replace Flux, the shared research computing cluster that currently serves over 300 research projects and 2,500 active users.

“Researchers will see improved performance, flexibility and reliability associated with newly purchased hardware, as well as changes in policies that will result in greater efficiencies and ease of use,” said Brock Palen, director of ARC-TS.

The Great Lakes cluster will be available to all researchers on campus for simulation, modeling, machine learning, data science, genomics, and more. The platform will provide a balanced combination of computing power, I/O performance, storage capability, and accelerators.

ARC-TS is in the process of procuring the cluster. Only minimal interruption to ongoing research is expected. A “Beta” cluster will be available to help researchers learn the new system before Great Lakes is deployed in the first half of 2019.

The Flux cluster is approximately 8 years old, although many of the individual nodes are newer. One of the benefits of replacing the cluster is to create a more homogeneous platform.

Based on extensive input from faculty and other stakeholders across campus, the new Great Lakes cluster will be designed to deliver similar services and capabilities as Flux, including the ability to accommodate faculty purchases of hardware, access to GPUs and large-memory nodes, and improved support for emerging uses such as machine learning and genomics. The cluster will consist of approximately 20,000 cores.

For more information, contact hpc-support@umich.edu, and see arc-ts.umich.edu/systems-services/greatlakes, where updates to the project will be posted.

Intro to GPU & CUDA Programming

By |

This workshop is an introduction to GPU programing for scientific and engineering applications. The basics of GPU architecture will be presented. Parallel programing strategies will be discussed followed by actual programing examples.

Please bring a laptop if would like to try the examples during the session, but it is not necessary, since the examples will be available for you to try later on the Flux computing cluster. Participants should be familiar with programming and how to use Flux.

Introduction to the Flux cluster and batch computing

By |

Overview

This workshop will provide a brief overview of the components of the Flux Cluster. The main body of the workshop will cover the resource manager and scheduler, creating submissions scripts to run jobs and the options available in them, and hands-on experience. By the end of the workshop, every participant should have created a submission script, submitted a job, tracked its progress, and collected its output. Participants will have several working examples from which to build their own submissions scripts in their own home directories.

Prerequisites

Obtain a user account on Flux. If you do not have a Flux user account, click here to go to the account application page at: https://arc-ts.umich.edu/fluxform/

Register for Duo authentication. See below for details.

This course assumes familiarity with the Linux command line as might be got from the CSCAR/ARC-TS workshop Introduction to the Linux Command Line. In particular, participants should understand how files and folders work, be able to create text files using the nano editor, be able to create and remove files and folders, and understand what input and output redirection are and how to use them.

click here to register

Instructor

Bennet Fauber
Scientific Applications Analyst
Advanced Research Computing — Technology Services

Bennet Fauber is a member of the technical team that administers the Flux cluster.

Materials

In-class handouts

Supplementary materials

Course Preparation

In order to participate successfully in the workshop exercises, you must have a Flux user account and be registered for the Duo authentication service. The user account allows you to log in to the cluster, create, compile, and test applications, and prepare Flux jobs for submission. Information about registering for Duo is at the ITS Two Factor Authentication web page.

Flux user account

A single Flux user account can be used to prepare and submit jobs using various allocations. If you already already possess a user account, you can use it for this course, you can skip to “Flux allocation” below. If not, please visit https://arc-ts.umich.edu/fluxform to obtain one. A user account is free to members of the University community. Please note that obtaining an account requires human processing, so be sure to do this at least two business days before class begins.

Flux allocation

We create an allocation for the workshop so you can run jobs on the cluster during the workshop and for one day after for those who would like additional practice. The workshop allocation is quite limited and is intended only to run examples to help you cement the details of job submission and management. If you already have an existing Flux allocation, you can use that, though if there are any issues with that allocation, we will ask you to use the workshop allocation.

Laptop Preparation

The classroom contains Windows computers, which require your uniqname and UMICH password to login, and which have all necessary software preloaded.

If you want to use a laptop for the course, you are welcome to do so. However, if there are problems connecting your laptop, you will be asked to switch to the provided computer for the class. We will try to help resolve issues after the workshop.

If you want to try to prepare your laptop for use in the class, please see our web page on Preparing your laptop to use Flux. If you have a Windows computer, we recommend that you install PuTTY/WinSCP from the Blue Disc web site. Macintosh computers typically have the minimum requirements installed as part of the system.

More help

Please use our class mailing list hpc101course@umich.edu for questions, comments, or to seek further assistance.

Advanced batch computing on the Flux cluster

By |

Overview

This course will cover some more advanced topics in cluster computing on the U-M Flux Cluster. Topics to be covered include a review of common parallel programming models and basic use of Flux; dependent and array scheduling; advanced troubleshooting and analysis using checkjob, qstat, and other tools; and parallel debugging and profiling of C and Fortran code, including logging, gdb (line-oriented debugging), ddt (GUI-based debugging) and map (GUI-based profiling) of MPI and OpenMP programs. We will issue you a temporary allocation to use for the course, or you can use your existing Flux allocations, if any.

Prerequisites

Introduction to Batch Computing on Flux or equivalent. This course assumes familiarity with the Linux command line, text editing on Linux, and a basic understanding of Flux including how to submit and track jobs.

click here to register

Instructors

Dr. Charles J Antonelli
Advocacy and Research Support
LSA Information Technology

Charles is a High Performance Computing Consultant in the Advocacy and Research Support Group of LSAIT at the University of Michigan, where he is responsible for high performance computing support and education, and is an advocate to the Departments of History and Communications. Prior to this, he built a parallel data ingestion component of a novel earth science data assimilation system, a secure packet vault, and worked on the No. 5 ESS Switch at Bell Labs in the 80s. He has taught courses in operating systems, distributed file systems, C++ programming, security, and database application design.

Mark Champe
Advocacy and Research Support
LSA Information Technology

Materials

Course Preparation

In order to participate successfully in the workshop exercises, you must have a Flux user account, a Flux allocation (one is created for each workshop), and be enrolled in Duo. The user account allows you to log in to the cluster, create, compile, and test applications, and prepare Flux jobs for submission. The allocation allows you to submit those jobs, executing those applications in parallel on the cluster and charging their resource use against the allocation. Duo is required to help authenticate you to the cluster.


Flux user account

A single Flux user account can be used to prepare and submit jobs using various allocations. If you already already possess a user account, you can use it for this course, you can skip to “Flux allocation” below. If not, please visit https://arc-ts.umich.edu/fluxform to obtain one. A user account is free to members of the University community. Please note that obtaining an account requires human processing, so be sure to do this at least two business days before class begins.


Flux allocation

We create an allocation for the workshop so you can run jobs on the cluster during the workshop and for one day after for those who would like additional practice. The workshop allocation is quite limited and is intended only to run examples to help you cement the details of job submission and management. If you already have an existing Flux allocation, you can use that, though if there are any issues with that allocation, we will ask you to use the workshop allocation.


Duo Authentication

Duo two-factor authentication is required to log in to the cluster. When logging in, you will need to type your UMICH password as well as authenticate through Duo in order to access Flux.

If you need to enroll in Duo, follow the instructions at Getting Started: How to Enroll in Duo.

Please enroll in Duo before you come to class.

Laptop Preparation

You do not need to bring your own laptop to class. The classroom contains Windows computers, which require your uniqname and UMICH password to login, and that have all necessary software pre-loaded.

If you want to use a laptop for the course, you are welcome to do so:  please see our web page on Preparing your laptop to use Flux. However, if there are problems connecting your laptop, you will be asked to switch to the provided computer for the class. We cannot stop to debug connection issues with personal or departmental laptops during the class.

More help

Please use our class mailing list hpc201course@umich.edu for questions, comments, or to seek further assistance.

Hadoop and Spark Workshop

By |

Overview

Learn how to process large amounts (up to terabytes) of data using SQL and/or simple programming models available in Python, R, Scala, and Java. Computers will be provided to follow along with hands-on examples; users can also bring laptops.

Prerequisites

Intro to the Linux Command Line or equivalent. This course assumes familiarity with the Linux command line.

A user account on Flux. If you do not have a Flux user account, click here to go to the account application page at: https://arc-ts.umich.edu/fluxform/

Duo authentication.

Duo two-factor authentication is required to log in to the cluster. When logging in, you will need to type your UMICH password as well as authenticate through Duo in order to access Flux.

If you need to enroll in Duo, follow the instructions at Getting Started: How to Enroll in Duo.

click here to register

Instructor

Brock Palen
Director
ARC-TS

Brock has over 10 years of high performance computing and data intensive computing experience in an academic environment. He currently works with the team at ARC-TS to provide HPC, Data Science, storage, and other research computing services to the University. Brock also is the NSF XSEDE projects Campus Champion representing the schools to this and other national computing infrastructures and organizations.

Materials

Course Preparation

In order to participate successfully in the class exercises, you must have a Flux user account. The user account allows you to log in to the cluster, create, compile, and test applications, and transfer data into Hadoop’s filesystem for processing.

Flux user account

A single Flux user account can be used to prepare and submit jobs using various allocations. If you already already possess a user account, you can use it for this course, you can skip to “Flux allocation” below. If not, please visit https://arc-ts.umich.edu/fluxform to obtain one. A user account is free to members of the University community. Please note that obtaining an account requires human processing, so be sure to do this at least two business days before class begins.

Duo Authentication

Duo two-factor authentication is required to log in to the cluster. When logging in, you will need to type your UMICH password as well as authenticate through Duo in order to access Flux.

If you need to enroll in Duo, follow the instructions at Getting Started: How to Enroll in Duo.

More help

Please email hpc-support@umich.edu for questions, comments, or to seek further assistance.

Advanced batch computing on the Flux cluster

By |

Overview

This course will cover some more advanced topics in cluster computing on the U-M Flux Cluster. Topics to be covered include a review of common parallel programming models and basic use of Flux; dependent and array scheduling; advanced troubleshooting and analysis using checkjob, qstat, and other tools; and parallel debugging and profiling of C and Fortran code, including logging, gdb (line-oriented debugging), ddt (GUI-based debugging) and map (GUI-based profiling) of MPI and OpenMP programs. We will issue you a temporary allocation to use for the course, or you can use your existing Flux allocations, if any.

Prerequisites

Introduction to Batch Computing on Flux or equivalent. This course assumes familiarity with the Linux command line, text editing on Linux, and a basic understanding of Flux including how to submit and track jobs.

click here to register

Instructors

Dr. Charles J Antonelli
Advocacy and Research Support
LSA Information Technology

Charles is a High Performance Computing Consultant in the Advocacy and Research Support Group of LSAIT at the University of Michigan, where he is responsible for high performance computing support and education, and is an advocate to the Departments of History and Communications. Prior to this, he built a parallel data ingestion component of a novel earth science data assimilation system, a secure packet vault, and worked on the No. 5 ESS Switch at Bell Labs in the 80s. He has taught courses in operating systems, distributed file systems, C++ programming, security, and database application design.

Mark Champe
Advocacy and Research Support
LSA Information Technology

Materials

Course Preparation

In order to participate successfully in the workshop exercises, you must have a Flux user account, a Flux allocation (one is created for each workshop), and be enrolled in Duo. The user account allows you to log in to the cluster, create, compile, and test applications, and prepare Flux jobs for submission. The allocation allows you to submit those jobs, executing those applications in parallel on the cluster and charging their resource use against the allocation. Duo is required to help authenticate you to the cluster.


Flux user account

A single Flux user account can be used to prepare and submit jobs using various allocations. If you already already possess a user account, you can use it for this course, you can skip to “Flux allocation” below. If not, please visit https://arc-ts.umich.edu/fluxform to obtain one. A user account is free to members of the University community. Please note that obtaining an account requires human processing, so be sure to do this at least two business days before class begins.


Flux allocation

We create an allocation for the workshop so you can run jobs on the cluster during the workshop and for one day after for those who would like additional practice. The workshop allocation is quite limited and is intended only to run examples to help you cement the details of job submission and management. If you already have an existing Flux allocation, you can use that, though if there are any issues with that allocation, we will ask you to use the workshop allocation.


Duo Authentication

Duo two-factor authentication is required to log in to the cluster. When logging in, you will need to type your UMICH password as well as authenticate through Duo in order to access Flux.

If you need to enroll in Duo, follow the instructions at Getting Started: How to Enroll in Duo.

Please enroll in Duo before you come to class.

Laptop Preparation

You do not need to bring your own laptop to class. The classroom contains Windows computers, which require your uniqname and UMICH password to login, and that have all necessary software pre-loaded.

If you want to use a laptop for the course, you are welcome to do so:  please see our web page on Preparing your laptop to use Flux. However, if there are problems connecting your laptop, you will be asked to switch to the provided computer for the class. We cannot stop to debug connection issues with personal or departmental laptops during the class.

More help

Please use our class mailing list hpc201course@umich.edu for questions, comments, or to seek further assistance.

Introduction to the Flux cluster and batch computing

By |

Overview

This workshop will provide a brief overview of the components of the Flux Cluster. The main body of the workshop will cover the resource manager and scheduler, creating submissions scripts to run jobs and the options available in them, and hands-on experience. By the end of the workshop, every participant should have created a submission script, submitted a job, tracked its progress, and collected its output. Participants will have several working examples from which to build their own submissions scripts in their own home directories.

Prerequisites

Obtain a user account on Flux. If you do not have a Flux user account, click here to go to the account application page at: https://arc-ts.umich.edu/fluxform/

Register for Duo authentication. See below for details.

This course assumes familiarity with the Linux command line as might be got from the CSCAR/ARC-TS workshop Introduction to the Linux Command Line. In particular, participants should understand how files and folders work, be able to create text files using the nano editor, be able to create and remove files and folders, and understand what input and output redirection are and how to use them.

click here to register

Instructor

Bennet Fauber
Scientific Applications Analyst
Advanced Research Computing — Technology Services

Bennet Fauber is a member of the technical team that administers the Flux cluster.

Materials

In-class handouts

Supplementary materials

Course Preparation

In order to participate successfully in the workshop exercises, you must have a Flux user account and be registered for the Duo authentication service. The user account allows you to log in to the cluster, create, compile, and test applications, and prepare Flux jobs for submission. Information about registering for Duo is at the ITS Two Factor Authentication web page.

Flux user account

A single Flux user account can be used to prepare and submit jobs using various allocations. If you already already possess a user account, you can use it for this course, you can skip to “Flux allocation” below. If not, please visit https://arc-ts.umich.edu/fluxform to obtain one. A user account is free to members of the University community. Please note that obtaining an account requires human processing, so be sure to do this at least two business days before class begins.

Flux allocation

We create an allocation for the workshop so you can run jobs on the cluster during the workshop and for one day after for those who would like additional practice. The workshop allocation is quite limited and is intended only to run examples to help you cement the details of job submission and management. If you already have an existing Flux allocation, you can use that, though if there are any issues with that allocation, we will ask you to use the workshop allocation.

Laptop Preparation

The classroom contains Windows computers, which require your uniqname and UMICH password to login, and which have all necessary software preloaded.

If you want to use a laptop for the course, you are welcome to do so. However, if there are problems connecting your laptop, you will be asked to switch to the provided computer for the class. We will try to help resolve issues after the workshop.

If you want to try to prepare your laptop for use in the class, please see our web page on Preparing your laptop to use Flux. If you have a Windows computer, we recommend that you install PuTTY/WinSCP from the Blue Disc web site. Macintosh computers typically have the minimum requirements installed as part of the system.

More help

Please use our class mailing list hpc101course@umich.edu for questions, comments, or to seek further assistance.

Introduction to the Linux Command Line

By |

This course will familiarize the student with the basics of accessing and interacting with Linux computers using the GNU/Linux operating system’s Bash shell, also generically referred to as “the command line”. Topics include: a brief overview of Linux, the Bash shell, navigating the file system, basic commands, shell redirection, permissions, processes, and the command environment. The workshop will also provide a quick introduction to nano a simple text editor that will be used in subsequent workshops to edit files.

click here to register

Instructor

Kenneth Weiss
IT Project Senior Manager
Medical School Information Services (MSIS)

Ken is a High Performance Computing Consultant in the Computational Research Consulting Division of MSIS at the University of Michigan. He works with a team of IT specialists to provide high performance computing support and training for the Medical School. Prior to this, he spent 21 years managing research computing, including an HPC cluster, for Dr. Charles Sing in the Human Genetics Department.

Materials

Course Preparation

You must register at least three full days prior to the event so that we have time to insure you have proper UM credentials for the workshop. This allows enough time for you to get your account adjusted by ITS in case you do not have access to the Linux systems.

Introduction to the Linux Command Line

By |

This course will familiarize the student with the basics of accessing and interacting with Linux computers using the GNU/Linux operating system’s Bash shell, also generically referred to as “the command line”. Topics include: a brief overview of Linux, the Bash shell, navigating the file system, basic commands, shell redirection, permissions, processes, and the command environment. The workshop will also provide a quick introduction to nano a simple text editor that will be used in subsequent workshops to edit files.

click here to register

Instructor

Kenneth Weiss
IT Project Senior Manager
Medical School Information Services (MSIS)

Ken is a High Performance Computing Consultant in the Computational Research Consulting Division of MSIS at the University of Michigan. He works with a team of IT specialists to provide high performance computing support and training for the Medical School. Prior to this, he spent 21 years managing research computing, including an HPC cluster, for Dr. Charles Sing in the Human Genetics Department.

Materials

Course Preparation

You must register at least three full days prior to the event so that we have time to insure you have proper UM credentials for the workshop. This allows enough time for you to get your account adjusted by ITS in case you do not have access to the Linux systems.

HPC training workshops begin Tuesday, Feb. 13

By | Educational, Events, General Interest, Happenings, HPC, News

series of training workshops in high performance computing will be held Feb. 12 through March 6, 2018, presented by CSCAR in conjunction with Advanced Research Computing – Technology Services (ARC-TS).

Introduction to the Linux command Line
This course will familiarize the student with the basics of accessing and interacting with Linux computers using the GNU/Linux operating system’s Bash shell, also known as the “command line.”
Location: East Hall, Room B254, 530 Church St.
Dates: (Please sign up for only one)
• Tuesday, Feb. 13, 1 – 4 p.m. (full descriptionregistration)
• Friday, Feb. 16, 9 a.m. – noon (full description | registration)

Introduction to the Flux cluster and batch computing
This workshop will provide a brief overview of the components of the Flux cluster, including the resource manager and scheduler, and will offer students hands-on experience.
Location: East Hall, Room B254, 530 Church St.
Dates: (Please sign up for only one)
• Monday, Feb. 19, 1 – 4 p.m. (full description | registration)
• Tuesday, March 6, 1 – 4 p.m. (full description | registration)

Advanced batch computing on the Flux cluster
This course will cover advanced areas of cluster computing on the Flux cluster, including common parallel programming models, dependent and array scheduling, and a brief introduction to scientific computing with Python, among other topics.
Location: East Hall, Room B250, 530 Church St.
Dates: (Please sign up for only one)
• Wednesday, Feb. 21, 1 – 5 p.m. (full description | registration)
• Friday, Feb. 23, 1 – 5 p.m. (full description | registration)

Hadoop and Spark workshop
Learn how to process large amounts (up to terabytes) of data using SQL and/or simple programming models available in Python, R, Scala, and Java.
Location: East Hall, Room B250, 530 Church St.
Dates: (Please sign up for only one)
• Thursday, Feb. 22, 1 – 5 p.m. (full description | registration)