By Sam Mitchell, Senior Solution Architect
Public cloud is an easy choice for enterprises looking to extend high performance workloads, reduce infrastructure costs and increase flexibility. The cloud offers the chance to reduce the capital cost of owning and managing excess compute capacity and storage for all workloads. Enterprises can avoid the hidden costs of unused compute capacity by “cloud bursting” or shifting some peak demand toward the cloud-based HPC grid extensions. But how do you connect to existing grid resources and attest to security compliance?
For the security and network management needs of HPC users considering the cloud, the best solution is connecting to the existing grid with overlay networks. An overlay network simply creates a private, sealed network on top of any existing network. Using overlay networks over top of public cloud resources can add the flexibility, high availability and the robust security that HPC grid operators need to cope with unforeseen capacity demands.
Best practices for HPC in the Public Cloud incorporate trust and security from the beginning.
With the following tried and true best practices for high performance computing (HPC) in the public cloud, enterprises - even in regulated industries such as healthcare and financial services - can manage a secure cloud-based HPC environment and still benefit from cloud’s economies of scale. Once HPC nodes are set up and secured in cloud, connecting between existing HPC grids and new deployments can be complex. By using a manageable and compliant cloud network topology, enterprises can ease the transition into cloud-based HPC.
The path to HPC in the public cloud starts by selecting trustworthy cloud providers and creating secure cloud deployments.
Historically, HPC environments have been expensive to own, manage and operate as entirely on-premise compute capacity. One reason this happens is that organisations often require extra compute resource for irregular one-off jobs containing sensitive data such as intellectual property. Cloud infrastructure is an excellent way to expand quickly for unexpected one-off projects.
HPC grid extensions can ensure one-off projects do not break the bank and, with added encryption from an on-premise grid to a cloud-based grid extension, that the projects comply with regulatory requirements. Ultimately, HPC cloud best practices can help an enterprise save capital costs, prevent vendor lock-in, conserve IT resources and prevent organisations from having to change HPC vendors.
HPC Best Practices in Action: US Mutual Fund
A large mutual fund based in Boston uses the elasticity of public cloud to compute financial metrics that never had been possible in their internal infrastructure. The large public cloud they selected had the required elements of capacity, on-demand flexibility, and pay-as-you-go pricing. But they also wanted added security and the agility to prevent vendor lock-in.
What the public cloud offered, on its own, could not provide the security and control needed for this financial institution to extend their existing HPC grids on the same datacentre-based network. The mutual fund required VLAN isolation to ensure customer traffic was separate from all other data traveling to and within the cloud. They also wanted to ensure resilient file storage and data validity beyond the cloud providers’ offerings.
Rather than rebuild their HPC grid, the mutual fund wanted to rapidly connect and scale up the public cloud and determined that the most efficient strategy was to use an overlay network. Their solution also included full end-to-end and data-in-motion encryption required to meet the financial industry data protection regulations. The overlay network allowed the new HPC workloads to act like the existing HPC grid network and pass internal and external security tests.
With an overlay network, the mutual fund securely burst into public cloud IaaS as a natural extension to their grid. The HPC grid extension also ensured all data-in-motion was encrypted from the on premise grid to the cloud-based grid extension. The mutual fund could then incorporate their cloud HPC results into on-demand reports for their clients.
Public cloud saved expensive physical servers from sitting idle. Best practices prevented vendor lock-in and saved IT teams from re-architecting or changing HPC vendors. Now, the mutual fund company uses public cloud infrastructure to create a secure and automated natural HPC grid extension in which they flex up their processing power in seconds and back down when no longer needed.