TL;DR: Researchers are creating new virtual-computing environments suitable for the most data-intensive applications, including pharmaceuticals development, financial transactions, and sophisticated geological modeling for the oil and gas industry. Concerns about performance scalability, data security, and reliability are being addressed through enhanced connections between private and public clouds.
Conventional wisdom is that cloud services aren't suitable for high-performance computing (HPC) applications such as scientific research and enterprise data management. They lack the bandwidth, security, and sophisticated analysis tools required to harvest knowledge from massive datasets, according to the experts.
Now new data-analysis techniques and enhanced cloud services are turning the conventional wisdom on its ear. Large enterprises are using the cloud for developing new drugs, finding new sources of energy, and other data-intensive applications.
You might think that recent high-profile data breaches would prevent companies from trusting their valuable intellectual property to the cloud. In fact, firms such as pharmaceuticals giant Bristol Myers Squibb are increasing their reliance on cloud services for core operations. At the recent ISC Cloud conference in Heidelberg, Germany, Amazon Web Services High Performance Cloud Computing executive David Pellerin described how Bristol Myers uses AWS for clinical trials. Inside HPC's Rich Brueckner reports on Pellerin's presentation in an October 6, 2014, article.
Other examples cited by AWS's Pellerin are the life sciences company Illumina uploading human gene sequences to an AWS-hosted BaseSpace (similar AWS-based research projects are underway at Pfizer and Novartis), as well as Australian oil and gas research company Stochastic Solutions' cloud-based simulation and modeling tools. The companies' willingness to place such sensitive data in the cloud shows their confidence in the security of the services, according to Pellerin.
Improving the cloud portability of massive datasets
One of the biggest obstacles to use of the cloud for high-performance applications is the difficulty in moving or recreating the complex virtual-cluster environments HPC applications require. Researchers at the National Institute of Advanced Industrial Science and Technology (AIST) Information Technology Research Institute have addressed this problem by developing a virtual-cluster computer in the institute's private cloud that is easily recreated on Amazon's EC2 commercial cloud. The technology is described in a September 8, 2014, article on the Phys.org site.
The goal of the AIST research is to use virtualization to separate the application-executing environment from the actual hardware, but without impinging performance, which often suffers in virtualized computing environments. By doing so the researchers hope to make high-performance computing resources available to small and medium-size organizations that previously couldn't afford the storage and other resources such research requires.
Another approach to HPC in the cloud is the overlay network created by a large Boston-based mutual fund that emulated the company's HPC grid by isolating all traffic between the cloud and its customers. This provided the financial firm with the security and control it required to ensure regulatory compliance. In an October, 15, 2014, article, HPCWire's Sam Mitchell reports on this network and other efforts by enterprises to extend their HPC operations to the cloud.
Many of the characteristics of a cloud-based HPC operation are available today on the BitCan cloud storage service. BitCan supports heterogeneous MySQL and MongoDB databases as well as Unix/Linux systems and files. Backups are easy to set up using a single console that requires no client-side installs or plugins. The service expands or contracts your storage needs on demand, and your data is encrypted at both the communication and storage layers.
Visit the BitCan site to create a free 30-day trial account. There's no reason for companies of any size to be prevented from realizing the cost, efficiency, reliability, and security of cloud storage services.