Explore ARCExplore ARC

ARC Director Sharon Broude Geva elected Chair of the Coalition for Academic Scientific Computation

By | HPC, News

Dr. Sharon Broude Geva, Director of Advanced Research Computing at the University of Michigan, has been elected Chair of the Coalition for Academic Scientific Computation (CASC) for 2019.

Founded in 1989, CASC advocates for the use of advanced computing technology to accelerate scientific discovery for national competitiveness, global security, and economic success. The organization’s members represent 87 institutions of higher education and national labs.

The chair position is one of four elected CASC executive officers. The officers work closely as a team with the director of CASC. The Chair is responsible for arranging and presiding over general CASC meetings and acts as an official representative of CASC.

Geva served as CASC secretary in 2015 and 2016, and vice-chair in 2017 and 2018.

The other executive officers for 2019 are Neil Bright, Georgia Institute of Technology, Vice Chair; Craig Stewart, Indiana University, Secretary; Scott Yockel, Harvard University, Treasurer; Rajendra Bose, Columbia University, past chair. Lisa Arafune is CASC Director.

 

Beta cluster available for learning Slurm; new scheduler to be part of upcoming cluster updates

By | Flux, General Interest, Happenings, HPC, News

New HPC resources to replace Flux and updates to Armis are coming.  They will run a new scheduling system (Slurm). You will need to learn the commands in this system and update your batch files to successfully run jobs. Read on to learn the details and how to get training and adapt your files.

In anticipation of these changes, ARC-TS has created the test cluster “Beta,” which will provide a testing environment for the transition to Slurm. Slurm will be used on Great Lakes; the Armis HIPAA-aligned cluster; and a new cluster called “Lighthouse” which will succeed the Flux Operating Environment in early 2019.

Currently, Flux and Armis use the Torque (PBS) resource manager and the Moab scheduling system; when completed, Great Lakes and Lighthouse will use the Slurm scheduler and resource manager, which will enhance the performance and reliability of the new resources. Armis will transition from Torque to Slurm in early 2019.

The Beta test cluster is available to all Flux users, who can login via ssh at ‘beta.arc-ts.umich.edu’. Beta has its own /home directory, so users will need to create or transfer any files they need, via scp/sftp or Globus.

Slurm commands will be needed to submit jobs. For a comparison of Slurm and Torque commands, see our Torque to Slurm migration page. For more information, see the Beta home page.

Support staff from ARC-TS and individual academic units will conduct several in-person and online training sessions to help users become familiar with Slurm. We have been testing Slurm for several months, and believe the performance gains, user communications, and increased reliability will significantly improve the efficiency and effectiveness of the HPC environment at U-M.

The tentative time frame for replacing or transitioning current ARC-TS resources is:

  • Flux to Great Lakes, first half of 2019
  • Armis from Torque to Slurm, January 2019
  • Flux Operating Environment to Lighthouse, first half of 2019
  • Open OnDemand on Beta, which replaces ARC Connect for web-based job submissions, Jupyter Notebooks, Matlab, and additional software packages, fall 2018

U-M selects Dell EMC, Mellanox and DDN to Supply New “Great Lakes” Computing Cluster

By | Flux, General Interest, Happenings, HPC, News

The University of Michigan has selected Dell EMC as lead vendor to supply its new $4.8 million Great Lakes computing cluster, which will serve researchers across campus. Mellanox Technologies will provide networking solutions, and DDN will supply storage hardware.

Great Lakes will be available to the campus community in the first half of 2019, and over time will replace the Flux supercomputer, which serves more than 2,500 active users at U-M for research ranging from aerospace engineering simulations and molecular dynamics modeling to genomics and cell biology to machine learning and artificial intelligence.

Great Lakes will be the first cluster in the world to use the Mellanox HDR 200 gigabit per second InfiniBand networking solution, enabling faster data transfer speeds and increased application performance.

“High-performance research computing is a critical component of the rich computing ecosystem that supports the university’s core mission,” said Ravi Pendse, U-M’s vice president for information technology and chief information officer. “With Great Lakes, researchers in emerging fields like machine learning and precision health will have access to a higher level of computational power. We’re thrilled to be working with Dell EMC, Mellanox, and DDN; the end result will be improved performance, flexibility, and reliability for U-M researchers.”

“Dell EMC is thrilled to collaborate with the University of Michigan and our technology partners to bring this innovative and powerful system to such a strong community of researchers,” said Thierry Pellegrino, vice president, Dell EMC High Performance Computing. “This Great Lakes cluster will offer an exceptional boost in performance, throughput and response to reduce the time needed for U-M researches to make the next big discovery in a range of disciplines from artificial intelligence to genomics and bioscience.”

The main components of the new cluster are:

  • Dell EMC PowerEdge C6420 compute nodes, PowerEdge R640 high memory nodes, and PowerEdge R740 GPU nodes
  • Mellanox HDR 200Gb/s InfiniBand ConnectX-6 adapters, Quantum switches and LinkX cables, and InfiniBand gateway platforms
  • DDN GRIDScaler® 14KX® and 100 TB of usable IME® (Infinite Memory Engine) memory

“HDR 200G InfiniBand provides the highest data speed and smart In-Network Computing acceleration engines, delivering HPC and AI applications with the best performance, scalability and efficiency,” said Gilad Shainer, vice president of marketing at Mellanox Technologies. “We are excited to collaborate with the University of Michigan, Dell EMC and DataDirect Networks, in building a leading HDR 200G InfiniBand-based supercomputer, serving the growing demands of U-M researchers.”

“DDN has a long history of working with Dell EMC and Mellanox to deliver optimized solutions for our customers. We are happy to be a part of the new Great Lakes cluster, supporting its mission of advanced research and computing. Partnering with forward-looking thought leaders as these is always enlightening and enriching,” said Dr. James Coomer, SVP Product Marketing and Benchmarks at DDN.

Great Lakes will provide significant improvement in computing performance over Flux. For example, each compute node will have more cores, higher maximum speed capabilities, and increased memory. The cluster will also have improved internet connectivity and file system performance, as well as NVIDIA Tensor GPU cores, which are very powerful for machine learning compared to prior generations of GPUs.

“Users of Great Lakes will have access to more cores, faster cores, faster memory, faster storage, and a more balanced network,” said Brock Palen, Director of Advanced Research Computing – Technology Services (ARC-TS).

The Flux cluster was created approximately 8 years ago, although many of the individual nodes have been added since then. Great Lakes represents an architectural overhaul that will result in better performance and efficiency. Based on extensive input from faculty and other stakeholders across campus, the new Great Lakes cluster will be designed to deliver similar services and capabilities as Flux, including the ability to accommodate faculty purchases of hardware, access to GPUs and large-memory nodes, and improved support for emerging uses such as machine learning and genomics.

ARC-TS will operate and maintain the cluster once it is built. Allocations of computing resources through ARC-TS include access to hundreds of software titles, as well as support and consulting from professional staff with decades of combined experience in research computing.

Updates on the progress of Great Lakes will be available at https://arc-ts.umich.edu/greatlakes/.

ARC-TS seeks pilot users for two new research storage services

By | General Interest, Happenings, HPC, News

Advanced Research Computing – Technology Services (ARC-TS) is seeking pilot users for two new research storage services.

The first, Locker, is group project storage focused on large data sets, and is available at a cost less than half that of current primary storage services. Locker still provides encryption, replication, snapshots, and workstation access. Example use cases for Locker are research projects in climate studies, genomics, imaging, and other data-intensive sciences.

The second service, Data Den, provides archive class storage for research data that is not actively used. As our lowest cost research storage offering, Data Den provides “cold storage” for massive amounts of data with 20 petabytes of encrypted and replicated capacity. Data Den allows researchers to preserve data between rounds of funding and management plans, and to free up space in more expensive primary storage by moving valuable, but not currently used, data.

Those interested in participating in the pilots should contact ARC-TS at hpc-support@umich.edu.

ARC-TS begins work on new “Great Lakes” cluster to replace Flux

By | Flux, Happenings, HPC, News

Advanced Research Computing – Technology Services (ARC-TS) is starting the process of creating a new, campus-wide computing cluster, “Great Lakes,” that will serve the broad needs of researchers across the University. Over time, Great Lakes will replace Flux, the shared research computing cluster that currently serves over 300 research projects and 2,500 active users.

“Researchers will see improved performance, flexibility and reliability associated with newly purchased hardware, as well as changes in policies that will result in greater efficiencies and ease of use,” said Brock Palen, director of ARC-TS.

The Great Lakes cluster will be available to all researchers on campus for simulation, modeling, machine learning, data science, genomics, and more. The platform will provide a balanced combination of computing power, I/O performance, storage capability, and accelerators.

ARC-TS is in the process of procuring the cluster. Only minimal interruption to ongoing research is expected. A “Beta” cluster will be available to help researchers learn the new system before Great Lakes is deployed in the first half of 2019.

The Flux cluster is approximately 8 years old, although many of the individual nodes are newer. One of the benefits of replacing the cluster is to create a more homogeneous platform.

Based on extensive input from faculty and other stakeholders across campus, the new Great Lakes cluster will be designed to deliver similar services and capabilities as Flux, including the ability to accommodate faculty purchases of hardware, access to GPUs and large-memory nodes, and improved support for emerging uses such as machine learning and genomics. The cluster will consist of approximately 20,000 cores.

For more information, contact hpc-support@umich.edu, and see arc-ts.umich.edu/systems-services/greatlakes, where updates to the project will be posted.

ConFlux cluster expands

By | General Interest, Happenings, HPC, News

ARC-TS has installed 15 new compute nodes into the ConFlux cluster. These nodes have the same 20 cores CPU as the original set, but with 256 GB of RAM instead of 128 GB. Neither the original nodes nor the newly added ones contain any GPUs

As a result, jobs should spend less time in queue, and users can be more liberal in their memory requirements.

HPC training workshops begin Tuesday, Feb. 13

By | Educational, Events, General Interest, Happenings, HPC, News

series of training workshops in high performance computing will be held Feb. 12 through March 6, 2018, presented by CSCAR in conjunction with Advanced Research Computing – Technology Services (ARC-TS).

Introduction to the Linux command Line
This course will familiarize the student with the basics of accessing and interacting with Linux computers using the GNU/Linux operating system’s Bash shell, also known as the “command line.”
Location: East Hall, Room B254, 530 Church St.
Dates: (Please sign up for only one)
• Tuesday, Feb. 13, 1 – 4 p.m. (full descriptionregistration)
• Friday, Feb. 16, 9 a.m. – noon (full description | registration)

Introduction to the Flux cluster and batch computing
This workshop will provide a brief overview of the components of the Flux cluster, including the resource manager and scheduler, and will offer students hands-on experience.
Location: East Hall, Room B254, 530 Church St.
Dates: (Please sign up for only one)
• Monday, Feb. 19, 1 – 4 p.m. (full description | registration)
• Tuesday, March 6, 1 – 4 p.m. (full description | registration)

Advanced batch computing on the Flux cluster
This course will cover advanced areas of cluster computing on the Flux cluster, including common parallel programming models, dependent and array scheduling, and a brief introduction to scientific computing with Python, among other topics.
Location: East Hall, Room B250, 530 Church St.
Dates: (Please sign up for only one)
• Wednesday, Feb. 21, 1 – 5 p.m. (full description | registration)
• Friday, Feb. 23, 1 – 5 p.m. (full description | registration)

Hadoop and Spark workshop
Learn how to process large amounts (up to terabytes) of data using SQL and/or simple programming models available in Python, R, Scala, and Java.
Location: East Hall, Room B250, 530 Church St.
Dates: (Please sign up for only one)
• Thursday, Feb. 22, 1 – 5 p.m. (full description | registration)

U-M fosters thriving artificial intelligence and machine learning research

By | General Interest, HPC, News, Research

Research using machine learning and artificial intelligence — tools that allow computers to learn about and predict outcomes from massive datasets — has been booming at the University of Michigan. The potential societal benefits being explored on campus are numerous, from on-demand transportation systems to self-driving vehicles to individualized medical treatments to improved battery capabilities.

The ability of computers and machines generally to learn from their environments is having transformative effects on a host of industries — including finance, healthcare, manufacturing, and transportation — and could have an economic impact of $15 trillion globally according to one estimate.

But as these methods become more accurate and refined, and as the datasets needed become bigger and bigger, keeping up with the latest developments and identifying and securing the necessary resources — whether that means computing power, data storage services, or software development — can be complicated and time-consuming. And that’s not to mention complying with privacy regulations when medical data is involved.

“Machine learning tools have gotten a lot better in the last 10 years,” said Matthew Johnson-Roberson, Assistant Professor of Engineering in the Department of Naval Architecture & Marine Engineering and the Department of Electrical Engineering and Computer Science. “The field is changing now at such a rapid pace compared to what it used to be. It takes a lot of time and energy to stay current.”

Diagram representing the knowledge graph of an artificial intelligence system, courtesy of Jason Mars, assistant professor, Electrical Engineering and Computer Science, U-M

Johnson-Roberson’s research is focused on getting computers and robots to better recognize and adapt to the world, whether in driverless cars or deep-sea mapping robots.

“The goal in general is to enable robots to operate in more challenging environments with high levels of reliability,” he said.

Johnson-Roberson said that U-M has many of the crucial ingredients for success in this area — a deep pool of talented researchers across many disciplines ready to collaborate, flexible and personalized support, and the availability of computing resources that can handle storing the large datasets and heavy computing load necessary for machine learning.

“The people is one of the reasons I came here,” he said. “There’s a broad and diverse set of talented researchers across the university, and I can interface with lots of other domains, whether it’s the environment, health care, transportation or energy.”

“Access to high powered computing is critical for the computing-intensive tasks, and being able to leverage that is important,” he continued. “One of the challenges is the data. A major driver in machine learning is data, and as the datasets get more and more voluminous, so does the storage needs.”

Yuekai Sun, an assistant professor in the Statistics Department, develops algorithms and other computational tools to help researchers analyze large datasets, for example, in natural language processing. He agreed that being able to work with scientists from many different disciplines is crucial to his research.

“I certainly find the size of Michigan and the inherent diversity that comes with it very stimulating,” he said. “Having people around who are actually working in these application areas helps guide the direction and the questions that you ask.”

Sun is also working on analyzing the potential discriminatory effects of algorithms used in decisions like whether to give someone a loan or to grant prisoners parole.

“If you use machine learning, how do you hold an algorithm or the people who apply it accountable? What does it mean for an algorithm to be fair?” he said. “Can you check whether this notion of non-discrimination is satisfied?”

Jason Mars, an assistant professor in the Electrical Engineering and Computer Science department and co-founder of a successful spinoff called Clinc, is applying artificial intelligence to driverless car technology and a mobile banking app that has been adopted by several large financial institutions. The app, called Finie, provides a much more conversational interface between users and their financial information than other apps in the field.

“There is going to be an expansion of the number of problems solved and number of contributions that are AI-based,” Mars said. He predicted that more researchers at U-M will begin exploring AI and ML as they understand the potential.

“It’s going to require having the right partner, the right experts, the right infrastructure, and the best practices of how to use them,” he said.

He added that U-M does a “phenomenal job” in supporting researchers conducting AI and ML research.

“The level of support and service is awesome here,” he said. “Not to mention that the infrastructure is state of the art. We stay relevant to the best techniques and practices out there.”

Advanced Research Computing at U-M, in part through resources from the university-wide Data Science Initiative, provides computing infrastructure, consulting expertise, and support for interdisciplinary research projects to help scientists conducting artificial intelligence and machine learning research.

For example, Consulting for Statistics, Computing and Analytics Research, an ARC unit, has several consultants on staff with expertise in machine learning and predictive analysis with large, complex, and heterogeneous data. CSCAR recently expanded capacity to support very large-scale machine learning using tools such as Google’s TensorFlow.

CSCAR consultants are available by appointment or on a drop-in basis, free of charge. See cscar.research.umich.edu or email cscar@umich.edu for more information.

CSCAR also provides workshops on topics in machine learning and other areas of data science, including sessions on Machine Learning in Python, and an upcoming workshop in March titled “Machine Learning, Concepts and Applications.”

The computing resources available to machine learning and artificial intelligence researchers are significant and diverse. Along with the campus-wide high performance computing cluster known as Flux, the recently announced Big Data cluster Cavium ThunderX will give researchers a powerful new platform for hosting artificial intelligence and machine learning work. Both clusters are provided by Advanced Research Computing – Technology Services (ARC-TS).

All allocations on ARC-TS clusters include access to software packages that support AI/ML research, including TensorFlow, Torch, and Spark ML, among others.

ARC-TS also operates the Yottabyte Research Cloud (YBRC), a customizable computing platform that recently gained the capacity to host and analyze data governed by the HIPAA federal privacy law.

Also, the Michigan Institute for Data Science (MIDAS) (also a unit of ARC) has supported several AI/ML projects through its Challenge Initiative program, which has awarded more than $10 million in research support since 2015.

For example, the Analytics for Learners as People project is using sensor-based machine learning tools to translate data on academic performance, social media, and survey data into attributes that will form student profiles. Those profiles will help link academic performance and mental health with the personal attributes of students, including values, beliefs, interests, behaviors, background, and emotional state.

Another example is the Reinventing Public Urban Transportation and Mobility project, which is using predictive models based on machine learning to develop on-demand, multi-modal transportation systems for urban areas.

In addition, MIDAS supports student groups involved in this type of research such as the Michigan Student Artificial Intelligence Lab (MSAIL) and the Michigan Data Science Team (MDST).

(A version of this piece appeared in the University Record.)

U-M wraps up successful SC17 conference

By | General Interest, Happenings, HPC, News

Several University of Michigan researchers and professional IT staff attended the Supercomputing 17 (SC17) conference in Denver from Nov. 12-17, participating in a number of different ways, including demonstrations, presentations and tutorials.

U-M participation included:

  • Matt McLean, a Big Data systems administrator with ARC-TS, served as a panelist at a session titled “The ARM Software Ecosystem: Are We There Yet?” (Slides)
  • Jeff Sica, a research database administrator with ARC-TS, helped lead a Birds of a Feather session titled “Containers in HPC.” (Slides)
  • Quentin Stout (EECS) and Christiane Jablonowski (CLASP) taught the “Parallel Computing 101” tutorial.
  • Shawn McKee, U-M Department of Physics, and OSiRIS Principal Investigator, demonstrated Object Storage and Caching for Science (network topology diagrams)
  • Eric Boyd, Director of Research Networks, presented on Research Networking at the University of Michigan at the U-M exhibit booth.
  • Simon Adorf, Ph.D. Candidate, Chemical Engineering Department, U-M, presented on Simple Data and Workflow Management with Signac and GPU-Accelerated Predictive Material Design at the U-M exhibit booth.
  • ARC sponsored a networking and career networking reception put on by Women in HPC. ARC Director Sharon Broude Geva spoke at the event.
  • Amy Liebowitz, a network architect at ITS, worked on SCINet, a high-capacity network created every year for the conference. Liebowitz was on the routing team, which is responsible for installing, configuring and supporting the high performance conference network. The Routing Team also coordinated external connectivity with commodity Internet and R&E WAN service providers.

U-M partners with Cavium on Big Data computing platform

By | Feature, General Interest, Happenings, HPC, News

A new partnership between the University of Michigan and Cavium Inc., a San Jose-based provider of semiconductor products, will create a powerful new Big Data computing cluster available to all U-M researchers.

The $3.5 million ThunderX computing cluster will enable U-M researchers to, for example, process massive amounts of data generated by remote sensors in distributed manufacturing environments, or by test fleets of automated and connected vehicles.

The cluster will run the Hortonworks Data Platform providing Spark, Hadoop MapReduce and other tools for large-scale data processing.

“U-M scientists are conducting groundbreaking research in Big Data already, in areas like connected and automated transportation, learning analytics, precision medicine and social science. This partnership with Cavium will accelerate the pace of data-driven research and opening up new avenues of inquiry,” said Eric Michielssen, U-M associate vice president for advanced research computing and the Louise Ganiard Johnson Professor of Engineering in the Department of Electrical Engineering and Computer Science.

“I know from experience that U-M researchers are capable of amazing discoveries. Cavium is honored to help break new ground in Big Data research at one of the top universities in the world,” said Cavium founder and CEO Syed Ali, who received a master of science in electrical engineering from U-M in 1981.

Cavium Inc. is a leading provider of semiconductor products that enable secure and intelligent processing for enterprise, data center, wired and wireless networking. The new U-M system will use dual socket servers powered by Cavium’s ThunderX ARMv8-A workload optimized processors.

The ThunderX product family is Cavium’s 64-bit ARMv8-A server processor for next generation Data Center and Cloud applications, and features high performance custom cores, single and dual socket configurations, high memory bandwidth and large memory capacity.

Alec Gallimore, the Robert J. Vlasic Dean of Engineering at U-M, said the Cavium partnership represents a milestone in the development of the College of Engineering and the university.

“It is clear that the ability to rapidly gain insights into vast amounts of data is key to the next wave of engineering and science breakthroughs. Without a doubt, the Cavium platform will allow our faculty and researchers to harness the power of Big Data, both in the classroom and in their research,” said Gallimore, who is also the Richard F. and Eleanor A. Towner Professor, an Arthur F. Thurnau Professor, and a professor both of aerospace engineering and of applied physics.

Along with applications in fields like manufacturing and transportation, the platform will enable researchers in the social, health and information sciences to more easily mine large, structured and unstructured datasets. This will eventually allow, for example, researchers to discover correlations between health outcomes and disease outbreaks with information derived from socioeconomic, geospatial and environmental data streams.

U-M and Cavium chose to run the cluster on Hortonworks Data Platform, which is based on open source Apache Hadoop. The ThunderX cluster will deliver high performance computer services for the Hadoop analytics and, ultimately, a total of three petabytes of storage space.

“Hortonworks is excited to be a part of forward-leading research at the University of Michigan exploring low-powered, high-performance computing,” said Nadeem Asghar, vice president and global head of technical alliances at Hortonworks. “We see this as a great opportunity to further expand the platform and segment enablement for Hortonworks and the ARM community.”