As with many facets of life, database benchmarking has several myths or “urban legends” that need summarily dispelled. So I’m going to write a few short blogs focusing one by one on some of these misunderstood database benchmarking issues. Note that I am not preaching that database benchmarking is a worthwhile task, because there are many who feel it’s not. In fact I recently read an excellent Forrester paper by Noel Yuhanna on the decreased value of standardized database benchmarks. But for those who do decide to experiment with or perform some standardized database benchmarks, you will want to avoid some common misconceptions and traps.

 

Let’s start by first examining the concepts and measurements of a standardized database benchmark. In its simplest form, a standardize database benchmark simply requires that a well defined workload is submitted to a database server and that measureable results shall be observed. These results are then to be applied to some complex mathematical formulas which yield comparable numbers (i.e. can be used for “apples to apples” comparisons).

 

So in the case of the TPC-C (www.tpc.org/tpcc), an older on-line transaction processing (OLTP) benchmark, the metric used to report Maximum Qualified Throughput (MQTh) is the tpm-C – which is the number of “new orders” processed per minute. There’s other concurrent database activity and an order is more than just a single database operation, thus tpm-C represents a true "business throughput" rather than just a discrete transaction execution rate. Thus all TPC-C test results should be reported in tpm-C. Yet most people seem to focus on TPS instead. And many are just looking at the entire database workload in calculating TPS rather than successfully completed “new orders”, so they are twice as wrong (i.e. wrong metric and calculating transaction throughput wrong). Yet most people seem intent on merely getting high TPS rates as the true measure of success.

 

I liken measuring standardized database benchmarks to something we all know fairly well – driving an automobile. We measure driving success and failures along several driving “business” metrics:

 

In short, we generally care about the user perceptions of the experience rather than the internal workings of the automobile engine. So things such as RPM and MPH are less important. That may seem odd – that engine work effort and velocity are not critical. But they are just internal mechanics or side effects of what we really care about (i.e. results).

 

TPS rates are much like RPM or MPH depending upon your viewpoint. Regardless, they are not the true measure. Either express in terms of tpm-C or something more useful from the real business perspective – such as average response time, which SLA’s often require.