[ Back to index ]
+Following our successful community submission to MLPerf inference v3.0, +we will set up new weekly conf-calls shortly - please stay tuned for more details!
+Please add your topics for discussion in the meeting notes +or via GitHub tickets.
+Please join our mailing list here.
+See our R&D roadmap for Q4 2022 and Q1 2023
+MLCommons is a non-profit consortium of 50+ companies that was originally created +to develop a common, reproducible and fair benchmarking methodology for new AI and ML hardware.
+MLCommons has developed an open-source reusable module called loadgen +that efficiently and fairly measures the performance of inference systems. +It generates traffic for scenarios that were formulated by a diverse set of experts from MLCommons +to emulate the workloads seen in mobile devices, autonomous vehicles, robotics, and cloud-based setups.
+MLCommons has also prepared several reference ML tasks, models and datasets +for vision, recommendation, language processing and speech recognition +to let companies benchmark and compare their new hardware in terms of accuracy, latency, throughput and energy +in a reproducible way twice a year.
+The first goal of this open automation and reproducibility taskforce is to +develop a light-weight and open-source automation meta-framework +that can make MLOps and DevOps more interoperable, reusable, portable, +deterministic and reproducible.
+We then use this automation meta-framework to develop plug&play workflows +for the MLPerf benchmarks to make it easier for the newcomers to run them +across diverse hardware, software and data and automatically plug in +their own ML tasks, models, data sets, engines, libraries and tools.
+Another goal is to use these portable MLPerf workflows to help students, researchers and +engineers participate in crowd-benchmarking and exploration of the design space tradeoffs +(accuracy, latency, throughput, energy, size, etc.) of their ML Systems from the cloud to the +edge using the mature MLPerf methodology while automating the submission +of their Pareto-efficient configurations to the open division of the MLPerf +inference benchmark.
+The final goal is to help end-users reproduce MLPerf results +and deploy the most suitable ML/SW/HW stacks in production +based on their requirements and constraints.
+This MLCommons taskforce is developing an open-source and technology-neutral +Collective Mind meta-framework (CM) +to modularize ML Systems and automate their benchmarking, optimization +and design space exploration across continuously changing software, hardware and data.
+CM is the second generation of the MLCommons CK workflow automation framework +that was originally developed to make it easier to reproduce research papers and validate them in the real world.
+As a proof-of-concept, this technology was successfully used to automate +MLPerf benchmarking and submissions +from Qualcomm, HPE, Dell, Lenovo, dividiti, Krai, the cTuning foundation and OctoML. +For example, it was used and extended by Arjun Suresh +with several other engineers to automate the record-breaking MLPerf inference benchmark submission for Qualcomm AI 100 devices.
+The goal of this group is to help users automate all the steps to prepare and run MLPerf benchmarks +across any ML models, data sets, frameworks, compilers and hardware +using the MLCommons CM framework.
+Here is an example of current manual and error-prone MLPerf benchmark preparation steps:
+
Here is the concept of CM-based automated workflows:
+
We have finished prototyping the new CM framework in summer 2022 based on the feedback of CK users +and successfully used it to modularize MLPerf and automate the submission of benchmarking results to the MLPerf inference v2.1. +See this tutorial for more details.
+We continue developing CM as an open-source educational toolkit +to help the community learn how to modularize, crowd-benchmark, optimize and deploy +Pareto-efficient ML Systems based on the mature MLPerf methodology and portable CM scripts - +please check the deliverables section to keep track of our community developments +and do not hesitate to join this community effort!
+See our R&D roadmap for Q4 2022 and Q1 2023
+HPCA'22 presentation "MLPerf design space exploration and production deployment"
+Tools:
+Google Drive (public access)
+This project is supported by MLCommons, OctoML +and many great contributors.
+ + + + + + + + + + + + + +Moved to https://github.com/ctuning/artifact-evaluation/blob/master/docs/checklist.md
+ + + + + + + + + + + + + +Moved to https://github.com/ctuning/artifact-evaluation/blob/master/docs/faq.md
+ + + + + + + + + + + + + +