Raghunath Nambiar and Reza Taheri look at the TPC-Virtualization Work Group and the TPC Technology Conference on Performance Evaluation and Benchmarking

Over the past two decades, the Transaction Processing Performance Council (TPC) has had a significant impact on the computing industry’s use of industry-standard benchmarks. For example, vendors and end-users rely on TPC benchmarks to provide real-world data that is backed by a stringent and independent review process. Vendors also use TPC benchmarks to demonstrate performance competitiveness for their existing products, and to improve/monitor the performance of products-under-development. Many buyers often use TPC benchmark results as points of comparison when purchasing new computing systems.

Since its inception in 1988, the TPC has developed nine different benchmarks, each addressing distinct requirements of industry demands. Current TPC benchmark standards include TPC-C, TPC-E and TPC-H, and in 2006, the TPC introduced a common pricing specification across all its benchmarks. Foreseeing the importance of energy efficiency in evaluating hardware, the TPC also approved a common energy specification across all its benchmarks in 2009, to help buyers understand the energy efficiencies of different computing systems. This energy specification, or TPC-Energy, adds another important dimension to the TPC’s existing performance and price metrics.

TPC-Virtualization Benchmark Initiative
The TPC’s most recent benchmark development effort is in the area of virtualization. The TPC-Virtualization Work Group was formed to respond to the growth in virtualization technology and capture some of the common uses of virtualization with database workloads in a rigorous and independent benchmark for virtual environments.

Most virtual benchmarks today cover consolidation cases; none are aimed at transaction processing or decision support applications, the traditional areas addressed by TPC benchmarks. These areas are a large segment of the application space being virtualized today.  The challenge is no longer successfully consolidating multiple applications on a single physical server.

The new frontier is virtualizing resource-intensive workloads, including those which are distributed across multiple physical servers.

There is a strong demand for the TPC to produce a new virtualization benchmark in the market segments that the TPC has been serving since its inception:

• Customers are asking for industry standard benchmarks that stress virtual environments with enterprise class, database-heavy workloads. These workloads are exactly the type of environments the TPC has addressed for the last two decades.
• Vendors want a database-centric benchmark for comparing their technologies against other vendors in terms of performance, price-performance and energy efficiency.
• Many vendors have implemented virtualization assist features in hardware, and have been asking for a standard benchmark for competitive comparisons, and to track internal product improvements.
• The TPC-Virtualization Work Group is currently defining a benchmark with the following design goals in mind:
• Incorporates a database-centric workload
• Features a resource-intensive workload that stresses the virtualization layer
• Emphasizes storage and networking I/O in a virtualized environment
• Has a timely development cycle

The TPC-Virtualization Work Group has made substantial progress in defining the framework of a new benchmark, and both major systems and software vendors are represented in this group. Organizations that are interested in influencing the TPC benchmarking development process are encouraged to become members. Additional information is available at TPC. 

The TPC Technology Conference on Performance Evaluation and Benchmarking
The technology landscape is continually evolving, challenging industry experts and researchers to develop innovative techniques to evaluate and benchmark computing systems. The TPC strives to develop new benchmarks to keep pace, and one vehicle for achieving this objective is the sponsorship of the Technology Conference on Performance Evaluation and Benchmarking (TPCTC). The TPCTC provides industry experts and researchers with a forum to present and debate novel ideas and methodologies in performance evaluation, measurement and characterization.

The first TPCTC was conducted in conjunction with the 35th International Conference on Very Large Data Bases (VLDB 2009).  The formation of the TPC-Virtualization Work Group was a direct result of this conference.

The second TPCTC will be collocated with the 36th International Conference on Very Large Data Bases (VLDB 2010) on September 17 in Singapore. The TPC is a co-sponsor of VLDB 2010. Researchers and industry experts are encouraged to submit novel ideas and methodologies in performance evaluation, measurement and characterization. Additional information about the conference is available here, and conference registration information is available here.