Long-time chronicler of benchmarking results BM Seer adds his voice to the chorus of Forrester critics Noel Yuhanna, Mike Gilpin and David D'Silva, who contend, "TPC Benchmarks Don\'t Matter Anymore."

Seer lets the Forrester group speak for themselves in his blog on the subject:

"Transaction Processing Performance Council (TPC) benchmarks, once widely accepted as the standard DBMS benchmark, are becoming obsolete. Why? First, all top-tier DBMS vendors such as IBM, Microsoft, Oracle, Sybase, and Teradata are delivering high performance and scalability to support most large workloads.

"Second, TPC benchmarks no longer reflect the complex workloads of today's real-world deployments. Third, customers that need high-end performance often prefer internal benchmarks to TPC benchmarks.

"Finally, virtualization, cloud computing, and database-as-a-service are changing the way customers deploy databases, and TPC does not address these architectures. Enterprise architects performing assessments should not waste their time on TPC benchmarks of top-tier DBMS products, and tech industry marketers and product managers should redirect the millions they spend on benchmark engineering toward automated tuning and performance optimization."

To which BM Seer adds, "I've been saying for years these things about the TPC-C benchmark!"

"Who will have the maturity to kill TPC-C," Seer asks rhetorically, then follows with comments from a Sun press release that notes, "It's well understood in the technical communities that TPC-C no longer represents current customer workloads since the transaction load that its models are made of are small, primitive and disconnected transactions. While this model was acceptable for the workloads of the late 1980s, it misses the mark."

Judging even from the still relatively sparse comments appended to this blog of Seer's, this subject will generate some energy in the months to come.

More Information

Forrester report

June 2005 Interview with Bruce Lindsay (IBM Fellow) [...read more...]

		fullsource
Print Email Current Profile Archive Blog Generate newsletter