Q&A: Benchmark
breakthrough
By Kate Evans-Correia, Senior News
Editor
02 Dec 2001, searchStorage
The Storage Performance Council (SPC), a consortium of
vendors dedicated to the pursuit of defining and standardizing
storage subsystem benchmarks, has announced the first
industry-standard storage benchmark for direct attach and
network storage subsystems. Supported by such companies as
Compaq, Dell, Hitachi, Sun and IBM, the benchmark standard is
expected to have a huge impact on the way users evaluate and
source storage systems.
SPC Chairman Leah Schoeb, a staff engineer for Sun
Microsystems, talks to searchStorage Senior Editor Kate
Evans-Correia about the SPC's efforts and why this benchmark
is so important to users.
 |
| SPC Chairman Leah
Schoeb | What
indication have you had from end users that a standard is even
necessary?
![]()
Schoeb:
Today end-users have nothing to quantitatively and fairly
compare the performance and price/performance of enterprise
storage systems. Many of the companies participating in and
contributing to the SPC do so because they have received input
from the market that's investing in the SPC essential.
The SPC is considering forming an Executive Board of
end-users to steer our direction moving forward - this will
formalize end-user support for our effort.
How are benchmarks going to benefit the
users?
![]()
Schoeb:
When they buy equipment, there's a lot of research involved.
They have to make sure that the information they use to
compare products and vendors is reliable, standardized and
independently scrutinized. What this allows us to give the
consumer is more solid, educated information in making their
decision.
When will it be available to the
user?
![]()
Schoeb:
We have approved the specification to be a standard in
September. We're working on an associated kit for vendors to
run. We would like to have it available the first part of next
year.
|
|
| "Ultimately, the
quality of this benchmark will convince end-users that
this is a viable standard and that is why we've taken
over four years and made sure that it's worthy of being
labeled an industry standard." Leah Schoeb |
|
|
Why has it taken so
long for vendors to embrace a benchmark standard?
![]()
Schoeb: The problem has
been the technical difficulty of constructing a world-class
enterprise storage benchmark and achieving consensus among the
major enterprise storage vendors on that benchmark. This has
never been done before in an industry standards body and,
after almost five years of work, we understand why. The SPC-1
benchmark introduces a level of sophistication and
completeness in storage benchmarking unrivaled in the history
of the computer industry.
How conclusive is
this benchmark standard?
![]()
Schoeb: The SPC has carefully surveyed
prior benchmark technology to insure that SPC-1 is a
breakthrough benchmark relevance, accuracy, and completeness.
In addition, SPC-1 ultimately demonstrates not only the exact
performance of the storage system but also validates that the
storage system is "enterprise-class" with tests for
sustainability, repeatability, and data persistence. A single
run of the SPC-1 benchmark produces summary results for
end-users and the press, but it also produces thousands of
(background) detailed statistics that impart a brutally
precise characterization of the quality of the storage under
test.
What kind of tests or situations does
your test cover?
![]()
Schoeb: SPC-1 is designed to simulate an
enterprise class multi-user environment indicative of
OLTP/database and e-mail server environments. In building the
SPC-1 benchmark the SPC collected I/O traces from dozens real
world environments and sculpted the benchmarks workload to
precisely echo the targeted environment.
How
will you convince users this is a viable standard?
![]()
Schoeb:
Ultimately, the quality of this benchmark will convince
end-users that this is a viable standard and that is why we've
taken over four years and made sure that it's worthy of being
labeled an industry standard. We know that several storage
manufacturers are preparing to release SPC-1 test results. The
presence of test results on real products will create vast
confidence in SPC-1. Initially, SPC is dependent on the press
and the analysts to get the word out and create credibility
for the benchmark.
What's the benefit to the
vendor to have a standard benchmark?
![]()
Schoeb: For the first time, he who
has the best technology will have the opportunity to credibly
differentiate himself in the market. The process of product
selection can, for the first time, really on quantitative,
apples-to-apples and relevant price/performance metrics.
Additionally, vendors will, for the first time, have a tool
that their engineers can use to clearly assess the quality of
their product designs in real-world environments that are
relevant to end-user purchasing decisions.
Can
you explain briefly the process to develop a benchmark
standard?
![]()
Schoeb: No. It's a huge and daunting task
evolving I/O trace analysis, workload generator design,
techniques for insuring test sponsors don't cheat, auditing
process, practices for fully disclosing the behavior of the
tested storage configuration, configuration restrictions,
consensus building, and on, and on, and on. There are only a
handful of people on earth capable of handling this task --
that's why we created the SPC.
What has been
the response of other standard organizations like FCIA and
SNIA?
![]()
Schoeb:
We're in discussion with other industry associations on
forming alliances that we are not at liberty to discus at this
time. We are very grateful to the Computer Measurement Group
(CMG) for providing us a venue for the launch at the SPC-1
benchmark at their annual conference in Anaheim California.
I noticed that EMC, one of the largest storage
vendors, wasn't on your list of participating vendors. Have
you received resistance from some companies?
![]()
Schoeb: Most major storage
vendors are actively participating in the SPC. There are some
companies that aren't membes yet because economic factors have
prevented them from joining as members, but they still support
the SPC. However, the contributions of resources and
technology by some members has been incredible. EMC was
initially a member of the SPC and for whatever reason they
decided not to participate. The SPC is a non-profit, vendor
neutral organization, that is open to all and when they are
ready to focus again on performance we would love to see them
back.
Will you be expanding your benchmark
standards to other areas in storage?
![]()
Schoeb: Yes, our next objectives are
to consider a NAS benchmark or a benchmark for sequential
access environments. Additionally, we plan to release in 2002
a variant of the SPC-1 benchmark kit that is the most powerful
and extensible general-purpose performance analysis tool ever
offered. This tool will become the standard in stress testing
and analysis labs through the computer industry.
For more information Featured
Topic on Benchmarks
The
Storage Performance Council
|
.gif){kind=spacer}
