This post is to summarize the paper and I definitely encourage you to read the full report over here: http://www.iqcloud.net/wp-content/uploads/2013/07/Cloud-Computing-Performance-A-Comparative-Analysis-of-5-Large-Cloud-IaaS….pdf
Objective of analysis study
The objective of the paper is determine the price-performance value of the cloud providers. Providing some valuable insight for customer when selecting their prefer cloud vendor.
Figure 1 – Principle of value proposition [figure from the paper]
Who are being compared
The study (done in June 2013) compared five large IaaS providers in the industry:
The tests were run for 3 times in 5 consecutive days: May 25, 2013 – May 29, 2013.
The most common size for cloud server, Medium Size (or equivalent / similar setup) was chosen from the 5 cloud vendors:
Figure 2 – Medium VM Spec [figure from the paper]
The tests used Unixbench 5.1.3 as benchmarking the performance of Linux OPS running on virtualized infrastructure, producing rating out of 10 stars. Details of Unixbench can be found here: https://code.google.com/p/byte-unixbench/
Two important pieces of info are collected:
- Performance: how well the provider scores on Unixbench, and how consistent the scores are.
- Price-Performance: after performance scores are established, we factor in cost to understand how much performance a user can expect on return for every amount of money spent, i.e., the value.
The performance result shows that Windows Azure provides the best performance and notably 3 times higher than AWS EC2 on average!
Figure 3 – Performance Only Result [figure from the paper]
Figure 4 – Average Unixbench Score, derived from Figure 3 [figure from the paper]
Retail hourly price of the cloud providers are captured (pay-as-you-go) basis on date of experiment.
Figure 5 – Pay-per-hour price [figure from the paper]
By taking each score and dividing by the price, we can get a relative price-to-performance score for each provider. And here are the score (The higher the score, the better):
Figure 6 – Price-Performance Result [figure from the paper]
CloudSpecs score is a further normalized value from Figure 6, taking the highest value to 100. And here’re the scores:
With the cloudspecs score, the ratio of each of the providers are formed as following
While acknowledging that Unixbench is just one test, customers may always consider other factors when selecting their cloud vendor.
To conclude, Amazon EC2 and Windows Azure offers the lowest price at $0.12 per hour. However, Windows Azure performs much better than EC2 in this experiment (approximately 3 times). The experiment also shows that Rackspace scores worst in term of price-performance.