Introducing New AWS Executors
Today we’re excited to announce the release of a suite of executor plugins for Covalent on Amazon Web Services (AWS), featuring new integrations to popular services including Amazon Elastic Cloud Compute (EC2), Amazon Elastic Compute Cloud (ECS), AWS Lambda, AWS Batch, and Amazon Braket. These executors empower Covalent users to easily dispatch workflow tasks from their local laptop to one or more of these services without deep knowledge of cloud infrastructure or operations. Users with a greater degree of cloud experience can also use Covalent to iterate over cloud resource types and get a better understanding of optimal configurations for container-based workloads.
| Plugin | Use Case |
|---|---|
 | General-purpose or high-compute workloads where users want to control the exact instance type, for instance, when benchmarking algorithms. |
 | Extremely low-compute and/or low-runtime tasks, such as data staging or event triggers. |
 | Low-compute or low-memory tasks suitable for AWS Fargate. Ideal for users with existing ECS clusters or granular requirements for batch computing. |
 | General-purpose or high-compute workloads. Suitable for users with highly parallelizable jobs with similar compute requirements. |
 | Hybrid quantum-classical workloads which require on-line simulators or quantum computers, such as quantum optimization and quantum machine learning. |
AWS offers a variety of compute services targeted towards users from different backgrounds and with different compute requirements. The five services mentioned here offer a range of options, spanning from serverless (Lambda, ECS Fargate) to server-based (EC2, Batch), from low-compute (Lambda) to high-compute (Batch), and from classical to quantum. With integrations to Amazon Simple Storage Service (S3), Amazon Elastic Container Registry (ECR), as well as Amazon CloudWatch, users can be sure that their tasks are properly checkpointed and logged throughout a workflow for a fully cloud-native experience.
When to Use Different Executors
Let’s explore some of the scenarios where users might want to use some of these plugins. The covalent-ec2-plugin allows users with specific compute requirements to deploy a task in a workflow to a particular instance type in a particular availability zone for maximum reproducibility. Under the hood, Covalent uses Terraform to dynamically provision the instance at runtime before running the task. When the task has exited, the instance is likewise deprovisioned immediately to save on cloud costs. This executor may be useful for users interested in benchmarking applications, or for users trying to understand the optimal instance type for a task before migrating it to a cluster-based backend.
The covalent-ecs-plugin offers a slightly different experience. Users with an existing ECS cluster can deploy serverless tasks by simply specifying the cluster name, the number of CPU cores, and the amount of memory. One of the major benefits of ECS Fargate is the low cost and high availability compared to server-based solutions, not to mention the native scalability of the underlying compute resources. This executor is optimal for low-compute and low-memory tasks, such as machine learning model validation or mildly intensive data transformation operations.
The covalent-awslambda-plugin allows users to interact with the AWS Lambda service. In the scenario where users do not want to provision an ECS cluster but still have low-compute tasks, they can run them directly on AWS Lambda. In a sense, this solution offers serverless cloud compute with the least amount of setup involved, so it can be appealing to users with less cloud experience. Since AWS Lambda charges by the millisecond rather than by the second, this executor is particularly well-suited for extremely short cloud-native tasks, such as data movement or event triggering.
Finally the covalent-awsbatch-plugin is particularly well-suited for users running large-scale batch experiments with variable compute requirements. Such tasks, including machine learning, optimization, and data analysis, would not fare well with the EC2 executor which would involve potentially unbounded instance provisioning. However, AWS Batch allows users to interact with a batch queue and compute environment. This plugin is recommended for users who are already power-users of AWS Batch, as well as newcomers interested in learning more about batch computing.
Quantum Computing with Covalent and AWS
Along with the classical set of executors, we are thrilled to announce the release of the covalent-braket-plugin. This new addition to our suite of plugins enables seamless execution of quantum-classical hybrid jobs on Amazon Braket, a fully managed quantum computing service. With the integration of Amazon Braket, users can now effortlessly explore and build quantum algorithms within their Covalent workflows, leveraging the power of quantum computing without the need for extensive infrastructure or operational knowledge. This powerful plugin not only makes quantum computing more accessible but also simplifies the development process, allowing researchers and developers to focus on solving complex problems and driving innovation in the field.
Next Steps
Ultimately, users can choose to use any or all of these executors in a single Jupyter notebook. Like with other Covalent executor plugins, it’s as simple as adding a decorator on top of workflow tasks:
To get started now with Covalent’s AWS Plugins, users can install them via
pip install covalent-aws-plugins[all]
or view the source code on GitHub. To learn more about Covalent, check out its homepage on GitHub, the official Covalent documentation, or interact with the team directly on Slack.
If you like what we’re doing, please show your support by starring ⭐️ our repository.