Section: Top Stories

Issue: 4310

Date: December 6, 2001

EMC Stands Back From Benchmark

By Tim Stammers

Benchmark warfare came to the storage industry yesterday as IBM Corp said EMC Corp's decision not to support a new standard is "another indication that EMC's proprietary, antiquated Symmetrix hardware architecture is showing its age." Not at all said EMC, it is just that the standard in no way represents the real world, and could deliver "horrendously" wrong results. The benchmark, called SPC-1, and designed to represent OLTP, email management and other application performance, was launched yesterday by an industry group called the Storage Performance Council. That group includes Hitachi Ltd and IBM among its members, but not EMC, which like several other suppliers left the SPC last year.

EMC's lack of support has left the standard somewhat crippled. Without EMC's consent, there will be no way of using the benchmark to compare its hardware to that of IBM and HDS. Given EMC's external RAID market share, which is more than the total of HDS and IBM Corp combined, that will be a major limitation. EMC joined the SPC in 1999, but left when it decided that the benchmark was heading in the "wrong direction," because it would represent what it says are highly randomized I/O operations. It said it will only support the standard if it shifts to something "more representative." "Potentially customers could get information that could be outright misleading," said Ken Steinhardt, EMC's director of technology analysis. "We'd want to see workloads that are representative of real application environments. A real, specific OLTP specific to a real application would be radically different [to SPC-1] in what it would show," he said. At the core of EMC's argument are the algorithms present in any high-end storage system which heuristically predict which data to load into high-speed cache memory in order to speed performance. Those are EMC's crown jewels, according to Steindhardt, but the test won't let them shine through because the data accessed in the tests is too random for the algorithms to predict.

Unlike others at IBM, Brian Smith, manager of I/O subsystems at the company's storage division steered clear of any suggestions that EMC's decision is related to real performance concerns. "EMC pulled out too early. All three of us [IBM, EMC and HDS] have large caches," said "We've spent the last 18 months designing a benchmark which represents real workloads. We had access to real customer data to address that very concern. The major effort has been to make sure caching algorithms are accounted for, and that the tests represent real cache use," he said. "IBM was also concerned early on that the benchmark would turn out too random. I have to concede to EMC that that would have been a very easy way to go, but we hung on in there and have come up with realistic cache patterns." Smith said IBM granted the TPC a license to use a patented IBM technique to generate a test using real customer data supplied by Sun Microsystems. "What EMC is saying would have been valid, if the test stayed random, but there are sequential reads and writes that have been designed in, carefully," Smith said. Although the specification has been posted on the web for the past year, he said: "Maybe EMC aren't aware of how this ended up." If EMC is right and the test is over-randomized and so does not allow caching algorithms to work properly, it will put more emphasis on the I/O performance and hardware architecture of whatever machine it is testing. The Symmetrix has come under mounting criticism this year because of its age, and because unlike HDS' Lighting storage array, it is based on four buses, and not a switch. EMC is sensitive to this criticism, and has repeatedly stressed the maturity of the Symmetrix, which it compares to architectures such as the OS/390 mainframe, Sun Solaris, or the AS/400. "in fact, it's the youngest of all the major corporate platforms in the world." The company has also told ComputerWire that the architecture "has some years left in it." Smith admitted that benchmark testing is a risky business for vendors. "You can come out top in ten in a row, and then lose on the eleventh because you've missed something." As a result, they are not to be taken lightly he said. And that might explain why in a tough year, EMC has not been the only vendor to abandon the SPC. Formed in 1999, the group at one stage numbered 22 members, but is now down to just 13. "Some left because they realized they simply aren't in the disk business," he said. "HP left, and that was a shocker, but Compaq continues to be a member."

tim.stammers@computerwire.com