In the minds of most readers "caveat emptor" evokes a cautious, perhaps defensive response, just as "snake oil" might. "Caveat emptor 2" is exactly the title Joerg Moellenkamp gives to his blog that takes on the subject of what he alleges is IBM's casual use of the expression relative Performance or "rPerf," an expression the company uses in comparing the performance of its various systems.

What drew Moellenkamp's attention to rPerf was IBM's use of the metric to compare the performance of its p5 p595 and p6 p595 systems. In that comparison IBM reported that the p5 p595 yielded 100,512 QphH@3000GB, and the p6 p595 yielded 156,537 QphH@3000GB in the TPC-H benchmark.

That comparison aroused skepticism in Moellenkamp, who proceeded to analyze the results. His finding is that the p5 595 with 1.9 GHz yields a value of 306.21 times the baseline performance derived from a pSeries 640 system. The p6 595 delivers 553.01 times the baseline performance, a difference of 81%, he notes.

The catch is, he contends, that if the p5 p595 yielded 100,512 QphH@3000GB, and the p6 p595 yielded 156,537 QphH@3000GB, calculating the difference results in just 56%: Somewhere there is 25% missing, he asserts.

Conceding that the p595 uses many more disks than the p6, Moellenkamp goes on to highlight some further observations:

• The new system uses AIX 6.1 instead of AIX 5.3. 6.1 is regarded as an important step in scaling on large systems.

• The p6 configuration uses twice the memory of the p5 configuration. TPC-H is known to be a very memory intensive benchmark, and the only good I/O is cached I/O he contends.

• The p6 provides PCIe instead of PCI-X, thus providing a much faster I/O

• The p5 config used a general purpose database, while the p6 config used a specialized DWH database

• The p6 config used 4 GB SAN instead of 2 GB FC.

• The p6 config used 667 MHz DRAM instead of 533 MHz

So, in view of all this, what is the merit of rPerf as a metric, he wonders, especially in view of IBM's own pronouncement on the measure:

"The rPerf model is not intended to represent any specific public benchmark results and should not be reasonably used in that way. The model simulates some of the system operations such as CPU, cache and memory. However, the model does not simulate disk or network I/O operations."

In view of this assertion, Moellenkamp draws his own conclusion, which is that "rPerf is vastly overscaling the performance for I/O intensive tasks. The difference is 25% less performance between estimated performance by rperf and real performance measured in the benchmark."

He adds, " ... when you get an offer of IBM using rPerf used in the sizing you should be cautious (in the sense of very cautious) if it's really matching your needs."

A number of Moellenkamp's readers take issue with his understanding of rPerf, which they register in the comments section of the blog.

More Information

Is It Per-core Performance or System Design that Really Makes a Difference? [...read more...]

		fullsource
Print Email Current Profile Archive Blog Generate newsletter