By Dave Simpson

At the Computer Measurement Group Conference this week in Anaheim, CA, the Storage Performance Council announced the SPC-1 benchmark, a standardized test to measure the performance of disk subsystems. According to Leah Schoeb, SPC chairperson and a staff engineer at Sun, the benchmark is designed for both disk array manufacturers and end users.

Manufacturers can use the results of the tests to compare (either publicly or internally) the performance of their arrays to their competitors'. End users can use the benchmark results to compare the performance of a variety of disk subsystems, which should make it easier for them to develop a "short list" of prospective suppliers. Schoeb says that IT organizations may also use the SPC-1 benchmarks to specify performance parameters in their RFPs.

The suite was developed by SPC members Adaptec, Compaq, Dell, The Evaluator Group, Hitachi, IBM, Ideas International, LSI Logic Storage Systems, NEC, Sun, Unisys and Veritas Software. Conspicuously absent from that list are EMC and Hewlett-Packard, both of which were originally members but subsequently dropped out.

As evidence of how difficult it is for such a diverse group of vendors and consultants to develop a standard benchmark, the SPC was formed in 1996, and the SPC-1 is the group's first deliverable. The benchmark was originally due a year ago.

The SPC process is modeled after the Transaction Processing Council and its benchmarks. Manufacturers first run the tests, and then the results are certified by an independent auditor and sent out for a 60-day peer review process. In the case of disputes, an SPC technical advisory board, consisting of five individuals, will arbitrate. Benchmark results will be posted on the SPC Web site (www.storageperformance.org), as well as on the Web sites of the SPC's "preferred analyst" firms -- The Evaluator Group and Ideas International.

Schoeb expects the first benchmark results to be posted in the first quarter of next year.

The benchmark is modeled after OLTP, Web server, database and email server environments and data streams, and is designed to test performance and price-performance. It is not designed to test features such as reliability, availability and serviceability. Basically, the benchmark plots response time against throughput.

Although the initial test is for OLTP-type applications, in the future the SPC plans to add benchmarks (including one for network-attached storage devices), as well as the ability for end users to vary the workload of the test code to more accurately simulate their own applications.

The SPC is a non-profit organization funded by its member companies. For more information, visit www.storageperformance.org, or call (650) 556-9384.