Fishworks engineer Brendan Gregg, who has written previously about Analytics in the Sun Storage 7000 Series, here blogs on its I/O latency features. I/O latency, he writes, is the time to service and respond to an I/O request. Given that clients are often waiting for this to complete, it is often the most interesting metric for performance analysis -- more so than IOPS or throughput. Since graphing average I/O latency can result in misleading results, Gregg chooses instead to provide I/O latency as a heat map.

Using DTrace to gather his detail from the operating system provides a sufficient resolution of data for plotting, and the plot itself shows the effect of turning on the flash-based SSD read cache. Latency drops.

The graph resulting from Gregg's testing back-end throughput on his storage systems, which he did by 'lighting up' disks one by one with a 128 Kbyte streaming read workload to their raw device, resulted in a "rainbow" that he likens to a pterodactyl in shape. Readers will absolutely need to click on the link above to let Gregg himself guide them through this particular visualization.

Suffice it to say here that the "beak," the "head" and the "wing" clear indications of differing levels of latency, raised many more questions than answers, Gregg concedes, pointing out that, had the findings been presented as average latency, these behaviors might have gone unnoticed.

"Latency heat maps," he concludes "have shown us many behaviors that we don't yet understand, which we need to spend more time with Analytics to figure out. It can be exciting to discover strange phenomenons for the first time, and it's very useful and practical as well - latency matters."

More Information

Sun Storage 7000 Unified Storage Systems

DTrace [...read more...]

		fullsource
Print Email Current Profile Archive Blog Generate newsletter