[Submitted on 30 Apr 2020]

Download PDF

Abstract: The booming successes of machine learning in different domains boost
industry-scale deployments of innovative AI algorithms, systems, and
architectures, and thus the importance of benchmarking grows. However, the
confidential nature of the workloads, the paramount importance of the
representativeness and diversity of benchmarks, and the prohibitive cost of
training a state-of-the-art model mutually aggravate the AI benchmarking
challenges. In this paper, we present a balanced AI benchmarking methodology
for meeting the subtly different requirements of different stages in developing
a new system/architecture and ranking/purchasing commercial off-the-shelf ones.
Performing an exhaustive survey on the most important AI domain-Internet
services with seventeen industry partners, we identify and include seventeen
representative AI tasks to guarantee the representativeness and diversity of
the benchmarks. Meanwhile, for reducing the benchmarking cost, we select a
benchmark subset to a minimum-three tasks-according to the criteria: diversity
of model complexity, computational cost, and convergence rate, repeatability,
and having widely-accepted metrics or not. We contribute by far the most
comprehensive AI benchmark suite-AIBench. The evaluations show AIBench
outperforms MLPerf in terms of the diversity and representativeness of model
complexity, computational cost, convergent rate, computation and memory access
patterns, and hotspot functions. With respect to the AIBench full benchmarks,
its subset shortens the benchmarking cost by 41%, while maintaining the primary
workload characteristics. The specifications, source code, and performance
numbers are publicly available from the web site
this http URL.