The Amazon Sustainability Data Initiative (ASDI), AWS donates cloud resources, technical support and access to scalable infrastructure and fast networks, and offers High Performance Computing (HPC) solutions to support simulations of short-term climate with the National Center for Atmospheric Research ( NCAR) Community Earth System Model Version 2 (CESM2) and the Whole Atmosphere Community Climate Model (WACCM). These are among the most demanding climate and atmosphere models in the world.
ASDI strives to accelerate sustainability research and innovation by minimizing the cost and time required to collect and analyze large sustainability data sets. The program supports innovators and researchers with the data, tools and technical know-how they need to advance sustainability. Working with ASDI, AWS and SilverLining, a non-profit organization dedicated to ensuring a safe climate, the National Center for Atmospheric Research (NCAR) will run an ensemble of 30 climate model simulations on AWS. The climatic runs simulate the earth system over the period from 2035 to 2070 under an average warming scenario. Similar simulations are being carried out by the UK’s Met Office using its UKESM1 Earth system model and by academic researchers using NASA’s GISS model, with all resulting datasets hosted and openly accessible through the AWS Open Data Program. The simulation work will demonstrate the ability to use cloud infrastructure to further develop climate models in support of robust scientific studies by researchers around the world, and aims to accelerate and democratize climate science.
AWS worked with the NCAR team to develop an architecture that supports this project and can support other weather and climate modeling workloads. The recommended architecture uses AWS ParallelCluster to create the computing environment shown in the following architecture diagram.
The simulations use Amazon EC2 with C5n.18xlarge instances that support Elastic Fabric Adapter (EFA). EFA enables the Message Passing Interface (MPI) with low latency, which is necessary for high-performance scaling of the climate simulations. The simulation data is initially stored on a shared storage system with Amazon FSx for Luster. FSx enables cost-effective, high-performance, and scalable storage to be shared between all nodes running the climate simulation. After the simulations are complete, the data is post-processed before being transferred to Amazon S3, where it is shared with the community.
Those interested in learning how to set up this computing environment can explore several AWS workshops that will guide you through the entire process. The AWS ParallelCluster workshop takes you step-by-step through installing AWS ParallelCluster, configuring the cluster, the process of submitting your first HPC job, and finally exiting the cluster when you’re done. Similarly to working with FSx for Luster, the High Performance File Systems Workshop can provide guidance on setting up FSx for Luster for the high performance shared storage system for all compute notes. The workshop also shows how to compare storage system performance and monitor related metrics in Amazon CloudWatch.
“As the world prepares for the United Nations Conference on Climate Change (COP26), this initiative provides technology leadership and funding to run these important models at scale on AWS and advance climate research,” said Kara Hurst, vice president, global sustainability at Amazon. “It is vital that we understand how the climate is changing and how this is affecting communities and businesses. Amazon is proud to partner with NCAR, one of the world’s leading climate modeling centers, and an innovative organization like SilverLining to accelerate progress and expand access to powerful climate research tools and data. “
Collaborating research teams will use the data generated by these simulations to study the effects on the earth and human systems – including agriculture, drought, floods and human health – in different parts of the world. These studies will improve understanding of short-term climate and climate intervention responses and accelerate progress on a time-sensitive problem facing humanity.
“Climate research is vital in places like Cape Town, where severe drought and other impacts are currently occurring and expected to get worse over the next few decades. Our ability to participate in climate research is limited by local computer resources and networks, ”said Dr. Chris Lennard, Senior Researcher in the Climate System Analysis Group (CSAG) at the University of Cape Town. “The open access to models and data sets, as well as cloud-based high-performance computing environments to work with, enable us to be equally engaged as researchers and to design and conduct studies to help people in our parts of the world who were inconceivable in the past. “
AWS funding is provided through the ASDI grant program, which supports experimentation and development for sustainability-related projects. The funding program is open to anyone who wants to create prototypes of software, tools for sustainability-related work, proof of concept for moving research workloads or open data sets to the cloud, or who want to train a broader community on how to use the cloud for sustainability workloads. Applications are welcome and available at https://amazonsdi.com/credit.
Learn more about the Amazon Sustainability Data Initiative and the AWS Open Data Program.
As a reminder, you can learn a lot from AWS HPC engineers by doing this HPC Tech Short YouTube Channel, and following the AWS HPC blog channel.