What you should know: I am currently working for Orbit Cloud Solutions as Cloud Advisor, but any posts on this blog reflect my own views and opinions only.
From time to time we see the need to evaluate the performance of systems running in the cloud with those on-premises. As a reference (mostly for myself) I created a list of tools for this purpose. Note that most of these tools are in no way specific to cloud environments.
Storage performance
Performance of cloud storage in most environments depend on some kind of storage tier and size of the allotted storage. For comparable results, make sure to pick similar configurations in terms of tier and storage size for your tests. And I’d strongly recommend to pick a similar VM shape as well.
fio
Probably the most common tool for measuring disk throughput and IOPS is fio. A very nice guide on using it for analysis can be found in the Google Cloud documentation.
The configuration for the benchmarking run is defined in a configuration file, this might look like this:
[global]
directory=${TEST_DIR}
size=100M
group_reporting=1
time_based
runtime=60s
ramp_time=2s
;ioengine=libaio
direct=1
verify=0
iodepth=64
stonewall
unlink=1
[write_throughput]
bs=1M
rw=write
[write_iops]
bs=4K
rw=randwrite
[read_throughput]
bs=1M
rw=read
[read_iops]
bs=4K
rw=randread
Network performance
Network performance in most cases is influenced by choice of VM shape, in most cases a larger number of vCPU leads to more available bandwidth. Depending on the CSP, there often are specific options to optimize networking, e.g. fastpath in Azure. Make sure to explore those options and understand how they affect the parts of network performance that is critical for your systems.
Tools for this task have been around for a long time, so go with one of those, e.g.
DB Performance
For evaluation of DB performance the gold standard would be to run your own workloads that reflect your specific requirements. If you are able to do this, perfect. All others will have to use some other means of performance and load testing. If migrating from an existing environment, you can run the benchmarks in into the current and the new environments and investigate any differences.
Since this not an uncommon need, a wide range of tools for DB benchmarks are available. Those might be a good starting point for your performance analysis.
One caveat: Do not underestimate the complexity and prerequisites for using some of the available tools. As an example, Oracle Real Application Testing will require expertise and solid preparation of data capture and replay to get meaningful results.
Swingbench for Oracle databases
Swingbench is one fairly popular tool for performance benchmarking and load testing of Oracle databases. It will generate its own testing data and has a couple of predefined workload scenarios to run. So these are synthetic workloads and it is up to you to pick the right scenarios to run or to define one yourself.
Download straight from Github.
pgbench for PostgreSQL
PostgreSQL comes with its own benchmarking tool, pgbench.
mysqlslap for MySQL
MySQL comes with its own load testing / emulation tool, mysqlslap.
Cloud Performance Testing Suites
Then there are full performance testing suites focusing on cloud.
CloudSuite
CloudSuite is showing performance for various typical scenarios (data analytics, web server, …).
PerfKitBenchmarker
PerfKit is the cloud performance testing suites beast bringing together lots of tools to run comparison tests for cloud environments. It already includes some of the tools mentioned before like the whole CloudSuite and is maintained by GCP.