AWS HPC Integrations

Amazon Web Services (AWS) is the world’s most comprehensive cloud platform, trusted by millions of customers across industries. From startups to global enterprises and government agencies, AWS provides on-demand solutions for compute, storage, networking, AI, analytics, and more. The platform empowers organizations to innovate faster, reduce costs, and scale globally with unmatched flexibility and reliability. With services like Amazon EC2 for compute, Amazon S3 for storage, SageMaker for AI/ML, and CloudFront for content delivery, AWS covers nearly every business and technical need. Its global infrastructure spans 120 availability zones across 38 regions, ensuring resilience, compliance, and security. Backed by the largest community of customers, partners, and developers, AWS continues to lead the cloud industry in innovation and operational expertise.

Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides secure, resizable compute capacity in the cloud. It is designed to make web-scale cloud computing easier for developers. Amazon EC2’s simple web service interface allows you to obtain and configure capacity with minimal friction. It provides you with complete control of your computing resources and lets you run on Amazon’s proven computing environment. Amazon EC2 delivers the broadest choice of compute, networking (up to 400 Gbps), and storage services purpose-built to optimize price performance for ML projects. Build, test, and sign on-demand macOS workloads. Access environments in minutes, dynamically scale capacity as needed, and benefit from AWS’s pay-as-you-go pricing. Access the on-demand infrastructure and capacity you need to run HPC applications faster and cost-effectively. Amazon EC2 delivers secure, reliable, high-performance, and cost-effective compute infrastructure to meet demanding business needs.

AWS Batch enables developers, scientists, and engineers to easily and efficiently run hundreds of thousands of batch computing jobs on AWS. AWS Batch dynamically provisions the optimal quantity and type of compute resources (e.g., CPU or memory optimized instances) based on the volume and specific resource requirements of the batch jobs submitted. With AWS Batch, there is no need to install and manage batch computing software or server clusters that you use to run your jobs, allowing you to focus on analyzing results and solving problems. AWS Batch plans, schedules, and executes your batch computing workloads across the full range of AWS compute services and features, such as AWS Fargate, Amazon EC2 and Spot Instances. There is no additional charge for AWS Batch. You only pay for the AWS resources (e.g. EC2 instances or Fargate jobs) you create to store and run your batch jobs.

Amazon FSx for Lustre is a fully managed service that provides high-performance, scalable storage for compute-intensive workloads. Built on the open-source Lustre file system, it offers sub-millisecond latencies, up to hundreds of gigabytes per second of throughput, and millions of IOPS, making it ideal for applications such as machine learning, high-performance computing, video processing, and financial modeling. FSx for Lustre integrates seamlessly with Amazon S3, allowing you to link file systems to S3 buckets. This integration enables transparent access and processing of S3 data from a high-performance file system, with the ability to import and export data between FSx for Lustre and S3. The service supports multiple deployment options, including scratch file systems for temporary storage and persistent file systems for long-term storage, as well as SSD and HDD storage types to optimize cost and performance based on workload requirements.

AWS Parallel Computing Service (AWS PCS) is a managed service that simplifies running and scaling high-performance computing workloads and building scientific and engineering models on AWS using Slurm. It enables the creation of complete, elastic environments that integrate computing, storage, networking, and visualization tools, allowing users to focus on research and innovation without the burden of infrastructure management. AWS PCS offers managed updates and built-in observability features, enhancing cluster operations and maintenance. Users can build and deploy scalable, reliable, and secure HPC clusters through the AWS Management Console, AWS Command Line Interface (AWS CLI), or AWS SDK. The service supports various use cases, including tightly coupled workloads like computer-aided engineering, high-throughput computing such as genomics analysis, accelerated computing with GPUs, and custom silicon like AWS Trainium and AWS Inferentia.

AWS Marketplace is a curated digital catalog that enables customers to discover, purchase, deploy, and manage third-party software, data products, AI agents, and services directly within the AWS ecosystem. It provides access to thousands of listings across categories like security, machine learning, business applications, and DevOps tools. With flexible pricing models such as pay-as-you-go, annual subscriptions, and free trials, AWS Marketplace simplifies procurement and billing by integrating costs into a single AWS invoice. It also supports rapid deployment with pre-configured software that can be launched on AWS infrastructure. This streamlined approach allows businesses to accelerate innovation, reduce time-to-market, and maintain better control over software usage and costs.

With Amazon DCV, you do not need to choose between responsiveness and image quality. The bandwidth-adaptive streaming protocol allows Amazon DCV to provide near real-time responsiveness for your applications without compromising on the accuracy of the image. Highly responsive streaming experience allows customers to run graphics-intensive applications remotely without the need for expensive dedicated workstations, or transferring large amounts of data from the cloud to client machines. On Linux systems, Amazon DCV also enables multiple sessions to share a single GPU, helping customers to save further on server infrastructure costs. Multi-OS compatibility and browser based access provides service builders a stable and flexible protocol for streaming applications with support for both cloud and on-premises usage. Amazon DCV supports both Windows and Linux remote environments and offers native clients for Windows, Linux, and MacOS as well as an HTML5 client for web browser access.

Amazon FSx makes it easy and cost effective to launch, run, and scale feature-rich, high-performance file systems in the cloud. It supports a wide range of workloads with its reliability, security, scalability, and broad set of capabilities. Amazon FSx is built on the latest AWS compute, networking, and disk technologies to provide high performance and lower TCO. And as a fully managed service, it handles hardware provisioning, patching, and backups -- freeing you up to focus on your applications, your end users, and your business.

The AWS Nitro System is the foundation for the latest generation of Amazon EC2 instances, enabling AWS to innovate faster, reduce costs for customers, and deliver enhanced security and new instance types. By reimagining virtualization infrastructure, AWS has offloaded functions such as CPU, storage, and networking virtualization to dedicated hardware and software, allowing nearly all server resources to be allocated to instances. This architecture comprises several key components: Nitro Cards, which offload and accelerate I/O for functions like VPC, EBS, and instance storage; the Nitro Security Chip, providing a minimized attack surface and prohibiting administrative access to eliminate human error and tampering; and the Nitro Hypervisor, a lightweight hypervisor that manages memory and CPU allocation, delivering performance nearly indistinguishable from bare metal. The Nitro System's modular design allows for rapid delivery of EC2 instance types.

Elastic Fabric Adapter (EFA) is a network interface for Amazon EC2 instances that enables customers to run applications requiring high levels of inter-node communications at scale on AWS. Its custom-built operating system (OS) bypass hardware interface enhances the performance of inter-instance communications, which is critical to scaling these applications. With EFA, High-Performance Computing (HPC) applications using the Message Passing Interface (MPI) and Machine Learning (ML) applications using NVIDIA Collective Communications Library (NCCL) can scale to thousands of CPUs or GPUs. As a result, you get the application performance of on-premises HPC clusters with the on-demand elasticity and flexibility of the AWS cloud. EFA is available as an optional EC2 networking feature that you can enable on any supported EC2 instance at no additional cost. Plus, it works with the most commonly used interfaces, APIs, and libraries for inter-node communications.

AWS ParallelCluster is an open-source cluster management tool that simplifies the deployment and management of High-Performance Computing (HPC) clusters on AWS. It automates the setup of required resources, including compute nodes, a shared filesystem, and a job scheduler, supporting multiple instance types and job submission queues. Users can interact with ParallelCluster through a graphical user interface, command-line interface, or API, enabling flexible cluster configuration and management. The tool integrates with job schedulers like AWS Batch and Slurm, facilitating seamless migration of existing HPC workloads to the cloud with minimal modifications. AWS ParallelCluster is available at no additional charge; users only pay for the AWS resources consumed by their applications. With AWS ParallelCluster, you can use a simple text file to model, provision, and dynamically scale the resources needed for your applications in an automated and secure manner.