News

Mar 24
0

Combat COVID-19 using newly available HPC resources: COVID-19 High Performance Computing Consortium

By | Feature, HPC, News, Research

COVID-19 High Performance Computing Consortium

On March 23, 3030 the White House announced the launch of a new partnership that aims to unleash U.S. supercomputing resources to fight COVID-19: the COVID-19 High Performance Computing Consortium. The goal of the Consortium is to bring together the Federal government, industry, and academic leaders to provide access to the world’s most powerful high-performance computing resources in support of COVID-19 research. The access to these resources has the potential to significantly advance the pace of scientific discovery in the fight to stop the virus.

To request access to resources of the COVID-19 HPC Consortium, you must prepare a description, no longer than two pages, of your proposed work. To ensure your request is directed to the appropriate resource(s), your description should include the following sections. Do not include any proprietary information in proposals, since your request will be reviewed by staff from a number of consortium sites. It is expected that teams who receive Consortium access will publish their results in the open scientific literature.

Learn more at https://covid19-hpc.mybluemix.net .

 

 

 

 

Oct 14
0

46 Peta-FLOPS computation of defects in solid crystals is a finalist in the highest prize for scientific computing

By | HPC, News, Research

From left: Sambit Das, Phani Motamarri and Vikram Gavini

A team led by Prof. Vikram Gavini (Professor of Mechanical Engineering and MICDE affiliate) and including Dr. Sambit Das (MICDE Fellow) and Dr. Phani Motamarri (Assistant Research Scientist and MICDE affiliate), is one of two finalists nominated for this year’s Gordon Bell Prize. The award, generally considered to be the highest honor of its kind, worldwide, recognizes outstanding achievement in high-performance computing. Gavini’s team has developed a methodology that combines advanced finite-element discretization methods for Density Functional Theory (DFT)1 with efficient computational methodologies and mixed precision strategies to achieve a 46 Peta-FLOPS2 sustained performance on 3,800 GPU nodes of the Summit supercomputer. Their work titled “Fast, scalable and accurate finite-element based ab initio calculations using mixed precision computing: 46 PFLOPS simulation of a metallic dislocation3 system” also involved Dr. Bruno Turcksin and Dr. Ying Wai Li from Oak Ridge National Laboratory, and Los Alamos National Laboratory, and Mr. Brent Leback from NVIDIA Corporation.

Electron density contour of pyramidal II screw dislocation system in Mg with 61,640 electrons (6,164 Mg atoms).

First principle calculation methods4 have been immensely successful in predicting a variety of material properties.  These calculations are prohibitively expensive as the computational complexity scales with the number of electrons in the system. Prof. Gavini’s research work is focussed on developing fast and accurate algorithms for Kohn-Sham5 density functional theory, a workhorse of first principle approaches that occupies a significant fraction of the world’s supercomputing resources. In the current work, Dr. Das, Dr. Motamarri and Prof. Gavini used recent developments in the computational framework for real-space DFT calculations using higher-order adaptive finite elements, and pioneered algorithmic advances in the solution of the governing equations, along with a clever parallel implementation that reduced the data access costs and communication bottlenecks. This resulted in fast, accurate and scalable large-scale DFT calculations that are an order of magnitude faster than existing widely used DFT codes. They demonstrated an unprecedented sustained performance of 46 Peta-FLOPS on a dislocation system containing ~100,000 electrons, which is the subject of the Gordon Bell nomination.

Past winners of the Gordon Bell Prize have typically been large teams working on grand challenge problems in astrophysics, climate science, natural hazard modeling, quantum physics, materials science and public health. The purpose of the award is to track the progress over time of parallel computing, with particular emphasis on rewarding innovation in applying high-performance computing to applications in science, engineering, and large-scale data analytics. If you are attending the SuperComputing’19 conference this year in Denver, you can learn more about Dr. Das, Dr. Motamarri and Dr. Gavini’s achievement at the Gordon Bell Prize finalists’ presentations on Wednesday, November 20, 2019, at 4:15 pm in rooms 205-207. 

Related Publication: S. Das, P. Motamarri, V. Gavini, B. Turcksin, Y. W. Li, and B. Leback. “Fast, Scalable and Accurate Finite-Element Based Ab initio Calculations Using Mixed Precision Computing: 46 PFLOPS Simulation of a Metallic Dislocation System.” To appear in SC’19 Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis, Denver, CO, November 17–22, 2019.

[1] Density functional theory (DFT) is a computational quantum mechanical modeling method used in physics, chemistry and materials science to investigate the electronic structure (or nuclear structure) (principally the ground state) of many-body systems, in particular atoms, molecules, and the condensed phases. https://en.wikipedia.org/wiki/Density_functional_theory.
[2] A PETAFLOP is a unit of computing speed equal to one thousand million million (1015) floating-point operations per second.
[3] In materials science, dislocations are line defects that exist in crystalline solids.
[4] First principle calculation methods use the principle of quantum mechanics to compute properties directly from basic physical quantities such as, e.g., mass and charge.
[5] W. Kohn, L. J. Sham, Self-consistent equations including exchange and correlation effects, Phys. Rev. 140(4A) (1965) A1133.

Mar 08
0

Women in HPC launches mentoring program

By | Educational, General Interest, HPC, News

Women in High Performance Computing (WHPC) has launched a year-round mentoring program, providing a framework for women to provide or receive mentorship in high performance computing. Read more about the program at https://womeninhpc.org/2019/03/mentoring-programme-2019/

WHPC was created with the vision to encourage women to participate in the HPC community by providing fellowship, education, and support to women and the organizations that employ them. Through collaboration and networking, WHPC strives to bring together women in HPC and technical computing while encouraging women to engage in outreach activities and improve the visibility of inspirational role models.

The University of Michigan has been recognized as one of the first Chapters in the new Women in High Performance Computing (WHPC) Pilot Program. Read more about U-M’s chapter at https://arc.umich.edu/whpc/
Nov 13
0

ARC Director Sharon Broude Geva re-elected Chair of the Coalition for Academic Scientific Computation

By | HPC, News

Dr. Sharon Broude Geva, Director of Advanced Research Computing at the University of Michigan, has been re-elected as Chair of the Coalition for Academic Scientific Computation (CASC) for 2020.

Founded in 1989, CASC advocates for the use of advanced computing technology to accelerate scientific discovery for national competitiveness, global security, and economic success. The organization’s members represent 87 institutions of higher education and national labs.

The chair position is one of four elected CASC executive officers. The officers work closely as a team with the director of CASC. The Chair is responsible for arranging and presiding over general CASC meetings and acts as an official representative of CASC.

Geva served as CASC secretary in 2015 and 2016, and vice-chair in 2017 and 2018.

 

Oct 22
0

Beta cluster available for learning Slurm; new scheduler to be part of upcoming cluster updates

By | Flux, General Interest, Happenings, HPC, News

New HPC resources to replace Flux and updates to Armis are coming.  They will run a new scheduling system (Slurm). You will need to learn the commands in this system and update your batch files to successfully run jobs. Read on to learn the details and how to get training and adapt your files.

In anticipation of these changes, ARC-TS has created the test cluster “Beta,” which will provide a testing environment for the transition to Slurm. Slurm will be used on Great Lakes; the Armis HIPAA-aligned cluster; and a new cluster called “Lighthouse” which will succeed the Flux Operating Environment in early 2019.

Currently, Flux and Armis use the Torque (PBS) resource manager and the Moab scheduling system; when completed, Great Lakes and Lighthouse will use the Slurm scheduler and resource manager, which will enhance the performance and reliability of the new resources. Armis will transition from Torque to Slurm in early 2019.

The Beta test cluster is available to all Flux users, who can login via ssh at ‘beta.arc-ts.umich.edu’. Beta has its own /home directory, so users will need to create or transfer any files they need, via scp/sftp or Globus.

Slurm commands will be needed to submit jobs. For a comparison of Slurm and Torque commands, see our Torque to Slurm migration page. For more information, see the Beta home page.

Support staff from ARC-TS and individual academic units will conduct several in-person and online training sessions to help users become familiar with Slurm. We have been testing Slurm for several months, and believe the performance gains, user communications, and increased reliability will significantly improve the efficiency and effectiveness of the HPC environment at U-M.

The tentative time frame for replacing or transitioning current ARC-TS resources is:

  • Flux to Great Lakes, first half of 2019
  • Armis from Torque to Slurm, January 2019
  • Flux Operating Environment to Lighthouse, first half of 2019
  • Open OnDemand on Beta, which replaces ARC Connect for web-based job submissions, Jupyter Notebooks, Matlab, and additional software packages, fall 2018
Oct 16
0

U-M selects Dell EMC, Mellanox and DDN to Supply New “Great Lakes” Computing Cluster

By | Flux, General Interest, Happenings, HPC, News

The University of Michigan has selected Dell EMC as lead vendor to supply its new $4.8 million Great Lakes computing cluster, which will serve researchers across campus. Mellanox Technologies will provide networking solutions, and DDN will supply storage hardware.

Great Lakes will be available to the campus community in the first half of 2019, and over time will replace the Flux supercomputer, which serves more than 2,500 active users at U-M for research ranging from aerospace engineering simulations and molecular dynamics modeling to genomics and cell biology to machine learning and artificial intelligence.

Great Lakes will be the first cluster in the world to use the Mellanox HDR 200 gigabit per second InfiniBand networking solution, enabling faster data transfer speeds and increased application performance.

“High-performance research computing is a critical component of the rich computing ecosystem that supports the university’s core mission,” said Ravi Pendse, U-M’s vice president for information technology and chief information officer. “With Great Lakes, researchers in emerging fields like machine learning and precision health will have access to a higher level of computational power. We’re thrilled to be working with Dell EMC, Mellanox, and DDN; the end result will be improved performance, flexibility, and reliability for U-M researchers.”

“Dell EMC is thrilled to collaborate with the University of Michigan and our technology partners to bring this innovative and powerful system to such a strong community of researchers,” said Thierry Pellegrino, vice president, Dell EMC High Performance Computing. “This Great Lakes cluster will offer an exceptional boost in performance, throughput and response to reduce the time needed for U-M researches to make the next big discovery in a range of disciplines from artificial intelligence to genomics and bioscience.”

The main components of the new cluster are:

  • Dell EMC PowerEdge C6420 compute nodes, PowerEdge R640 high memory nodes, and PowerEdge R740 GPU nodes
  • Mellanox HDR 200Gb/s InfiniBand ConnectX-6 adapters, Quantum switches and LinkX cables, and InfiniBand gateway platforms
  • DDN GRIDScaler® 14KX® and 100 TB of usable IME® (Infinite Memory Engine) memory

“HDR 200G InfiniBand provides the highest data speed and smart In-Network Computing acceleration engines, delivering HPC and AI applications with the best performance, scalability and efficiency,” said Gilad Shainer, vice president of marketing at Mellanox Technologies. “We are excited to collaborate with the University of Michigan, Dell EMC and DataDirect Networks, in building a leading HDR 200G InfiniBand-based supercomputer, serving the growing demands of U-M researchers.”

“DDN has a long history of working with Dell EMC and Mellanox to deliver optimized solutions for our customers. We are happy to be a part of the new Great Lakes cluster, supporting its mission of advanced research and computing. Partnering with forward-looking thought leaders as these is always enlightening and enriching,” said Dr. James Coomer, SVP Product Marketing and Benchmarks at DDN.

Great Lakes will provide significant improvement in computing performance over Flux. For example, each compute node will have more cores, higher maximum speed capabilities, and increased memory. The cluster will also have improved internet connectivity and file system performance, as well as NVIDIA Tensor GPU cores, which are very powerful for machine learning compared to prior generations of GPUs.

“Users of Great Lakes will have access to more cores, faster cores, faster memory, faster storage, and a more balanced network,” said Brock Palen, Director of Advanced Research Computing – Technology Services (ARC-TS).

The Flux cluster was created approximately 8 years ago, although many of the individual nodes have been added since then. Great Lakes represents an architectural overhaul that will result in better performance and efficiency. Based on extensive input from faculty and other stakeholders across campus, the new Great Lakes cluster will be designed to deliver similar services and capabilities as Flux, including the ability to accommodate faculty purchases of hardware, access to GPUs and large-memory nodes, and improved support for emerging uses such as machine learning and genomics.

ARC-TS will operate and maintain the cluster once it is built. Allocations of computing resources through ARC-TS include access to hundreds of software titles, as well as support and consulting from professional staff with decades of combined experience in research computing.

Updates on the progress of Great Lakes will be available at https://arc-ts.umich.edu/greatlakes/.

Sep 26
0

ARC-TS seeks pilot users for two new research storage services

By | General Interest, Happenings, HPC, News

Advanced Research Computing – Technology Services (ARC-TS) is seeking pilot users for two new research storage services.

The first, Locker, is group project storage focused on large data sets, and is available at a cost less than half that of current primary storage services. Locker still provides encryption, replication, snapshots, and workstation access. Example use cases for Locker are research projects in climate studies, genomics, imaging, and other data-intensive sciences.

The second service, Data Den, provides archive class storage for research data that is not actively used. As our lowest cost research storage offering, Data Den provides “cold storage” for massive amounts of data with 20 petabytes of encrypted and replicated capacity. Data Den allows researchers to preserve data between rounds of funding and management plans, and to free up space in more expensive primary storage by moving valuable, but not currently used, data.

Those interested in participating in the pilots should contact ARC-TS at hpc-support@umich.edu.

Apr 18
0

ARC-TS begins work on new “Great Lakes” cluster to replace Flux

By | Flux, Happenings, HPC, News

Advanced Research Computing – Technology Services (ARC-TS) is starting the process of creating a new, campus-wide computing cluster, “Great Lakes,” that will serve the broad needs of researchers across the University. Over time, Great Lakes will replace Flux, the shared research computing cluster that currently serves over 300 research projects and 2,500 active users.

“Researchers will see improved performance, flexibility and reliability associated with newly purchased hardware, as well as changes in policies that will result in greater efficiencies and ease of use,” said Brock Palen, director of ARC-TS.

The Great Lakes cluster will be available to all researchers on campus for simulation, modeling, machine learning, data science, genomics, and more. The platform will provide a balanced combination of computing power, I/O performance, storage capability, and accelerators.

ARC-TS is in the process of procuring the cluster. Only minimal interruption to ongoing research is expected. A “Beta” cluster will be available to help researchers learn the new system before Great Lakes is deployed in the first half of 2019.

The Flux cluster is approximately 8 years old, although many of the individual nodes are newer. One of the benefits of replacing the cluster is to create a more homogeneous platform.

Based on extensive input from faculty and other stakeholders across campus, the new Great Lakes cluster will be designed to deliver similar services and capabilities as Flux, including the ability to accommodate faculty purchases of hardware, access to GPUs and large-memory nodes, and improved support for emerging uses such as machine learning and genomics. The cluster will consist of approximately 20,000 cores.

For more information, contact hpc-support@umich.edu, and see arc-ts.umich.edu/systems-services/greatlakes, where updates to the project will be posted.

Mar 06
1

ConFlux cluster expands

By | General Interest, Happenings, HPC, News

ARC-TS has installed 15 new compute nodes into the ConFlux cluster. These nodes have the same 20 cores CPU as the original set, but with 256 GB of RAM instead of 128 GB. Neither the original nodes nor the newly added ones contain any GPUs

As a result, jobs should spend less time in queue, and users can be more liberal in their memory requirements.