Skip to content

Commit

Permalink
v1.6.1 release
Browse files Browse the repository at this point in the history
  • Loading branch information
@gfursin
gfursin committed Feb 2, 2024
1 parent 4b0e6e5 commit c5ea901
Show file tree
Hide file tree
Showing 2 changed files with 86 additions and 36 deletions.
120 changes: 85 additions & 35 deletions cm/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,16 +9,49 @@

### About

[Collective Mind (CM)](https://github.com/mlcommons/ck) is a human-friendly interface to help everyone run, manage and reuse
a [growing number of MLPerf, MLOps and DevOps scripts](https://github.com/mlcommons/ck/tree/master/docs/list_of_scripts.md)
from [MLCommons projects](https://mlcommons.org) and [research papers](https://cTuning.org/ae)
in a unified way on any operating system with any software and hardware
either natively or inside containers.

Here are a few most commonly used examples from the CM users
that should run in the same way on Linux, MacOS, Windows and other platforms
(see [Getting Started Guide](docs/getting-started.md) to understand
how they work and how to reuse them in your projects):
**Collective Mind (CM)** is a [community project](https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md) to develop
a [collection of portable, extensible, technology-agnostic and ready-to-use automation recipes
with a human-friendly interface (aka CM scripts)](https://github.com/mlcommons/ck/tree/master/docs/list_of_scripts.md)
that automate all the manual steps required to build, run, benchmark and optimize complex ML/AI applications on any platform
with any software and hardware.

CM scripts are being developed based on the feedback from
[MLCommons engineers and researchers](https://github.com/mlcommons/ck/blob/master/docs/taskforce.md)
to help them assemble, run, benchmark and optimize complex AI/ML applications
across diverse and continuously changing models, data sets, software and hardware
from Nvidia, Intel, AMD, Google, Qualcomm, Amazon and other vendors.
They require Python 3.7+ with minimal dependencies and can run natively on Ubuntu, MacOS, Windows, RHEL, Debian, Amazon Linux
and any other operating system, in a cloud or inside automatically generated containers.

Some key requirements for the CM design are:
* must be non-intrusive and easy to debug, require zero changes to existing projects and must complement,
reuse, wrap and interconnect all existing automation scripts and tools (such as cmake, ML workflows,
python poetry and containers) rather than substituting them;
* must have a very simple and human-friendly command line with a Python API and minimal dependencies;
* must require minimal or zero learning curve by using plain Python, native scripts, environment variables
and simple JSON/YAML descriptions instead of inventing new languages;
* must run in a native environment with Ubuntu, Debian, RHEL, Amazon Linux, MacOS, Windows
and any other operating system while automatically generating container snapshots
with CM recipes for repeatability and reproducibility.

Below you can find a few examples of this collaborative engineering effort sponsored
by [MLCommons (non-profit organization with 125+ organizations)](https://mlcommons.org) -
a few most-commonly used [automation recipes](https://github.com/mlcommons/ck/tree/master/docs/list_of_scripts.md)
that can be chained into more complex automation workflows [using simple JSON or YAML](https://github.com/mlcommons/ck/blob/master/cm-mlops/script/app-image-classification-onnx-py/_cm.yaml).

You can try them yourself (you only need Python 3.7+, PIP, git and wget installed and optionally Docker if you want to
run CM scripts via automatically-generated containers - check the [installation guide](docs/installation.md) for more details).

*Note that the Collective Mind concept is to continue improving portability and functionality
of all CM automation recipes across rapidly evolving models, data sets, software and hardware
based on collaborative testing and feedback - don't hestiate to report encountered issues
[here](https://github.com/mlcommons/ck/issues) and/or contact us via [public Discord Server](https://discord.gg/JjWNWXKxwT)
to help this community effort!*


#### CM human-friendly command line

<sup>

```bash
pip install cmind
Expand All @@ -33,6 +66,7 @@ cmr "python app image-classification onnx" --input=computer_mouse.jpg
cmr "python app image-classification onnx" --input=computer_mouse.jpg --debug

cm find script "python app image-classification onnx"
cm load script "python app image-classification onnx" --yaml

cmr "get python" --version_min=3.8.0 --name=mlperf-experiments
cmr "install python-venv" --version_max=3.10.11 --name=mlperf
Expand Down Expand Up @@ -64,14 +98,19 @@ cm docker script "get coco dataset _val _2017" --to=d:\Downloads\COCO-2017-val -
cmr "run common mlperf inference" --implementation=nvidia --model=bert-99 --category=datacenter --division=closed
cm find script "run common mlperf inference"

cmr "get generic-python-lib _package.torch" --version=2.1.1
cmr "get generic-python-lib _package.torch" --version=2.1.2
cmr "get generic-python-lib _package.torchvision" --version=0.16.2
cmr "python app image-classification torch" --input=computer_mouse.jpg

cm rm repo mlcommons@ck
cm pull repo --url=https://zenodo.org/records/10581696/files/cm-mlops-repo-20240129.zip

cmr "install llvm prebuilt" --version=17.0.6
cmr "app image corner-detection"

cm run experiment --tags=tuning,experiment,batch_size -- echo --batch_size={{VAR1{range(1,8)}}}
cm replay experiment --tags=tuning,experiment,batch_size

cmr "get conda"

cm pull repo ctuning@cm-reproduce-research-projects
Expand All @@ -80,39 +119,50 @@ cmr "reproduce paper micro-2023 victima _run"

```

</sup>


#### CM unified Python API

<sup>

```python
import cmind
output=cmind.access({'action':'run', 'automation':'script',
'tags':'python,app,image-classification,onnx',
'input':'computer_mouse.jpg'})
if output['return']==0: print (output)
```
</sup>

</details>


#### Examples of modular containers and GitHub actions with CM commands

<small>

* https://github.com/mlcommons/inference/blob/master/.github/workflows/test-bert.yml
* https://github.com/mlcommons/ck/blob/master/cm-mlops/script/app-mlperf-inference/dockerfiles/bert-99.9/ubuntu_22.04_python_onnxruntime_cpu.Dockerfile

</small>


[CM scripts](https://github.com/mlcommons/ck/blob/master/docs/list_of_scripts.md)
were successfully used to [modularize MLPerf inference benchmarks](https://github.com/mlcommons/ck/blob/master/docs/mlperf/inference/README.md)
and help the community automate more than 95% of all performance and power submissions in the v3.1 round
across more than 120 system configurations (models, frameworks, hardware)
while reducing development and maintenance costs.

Besides automating MLCommons projects, the community also started started using
and extending [CM scripts](https://github.com/mlcommons/ck/tree/master/docs/list_of_scripts.md)
to modularize, run and benchmark other software projects and make it
easier to rerun, reproduce and reuse [research projects from published papers
at Systems and ML conferences]( https://cTuning.org/ae/micro2023.html ).

Collective Mind is a community project being developed by the [MLCommons Task Force on Automation and Reproducibility](https://github.com/mlcommons/ck/blob/master/docs/taskforce.md)
with great help from [MLCommons (70+ AI organizations)](https://mlcommons.org/),
[research community]( https://www.youtube.com/watch?v=7zpeIVwICa4 )
and [individual contributors](https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md) -
we want to have a common, non-intrusive, technology-agnostic, portable and easily-extensible interface
that requires minimal learning curve to start automating all manual and repetitive tasks including
downloading artifacts, installing tools, resolving dependencies,
running experiments, processing logs, and reproducing results.

That is why we implemented CM as a [small Python library](https://github.com/mlcommons/ck/tree/master/cm)
with minimal dependencies (Python 3.7+, git, wget), simple Python API and human-friendly command line
that simply searches for CM scripts by tags in all pulled Git repositories, automatically generates command lines
for a given script or tool on a given platform, updates all paths and environment variables,
runs a given automation either natively or inside automatically-generated containers
and unifies input and output as a Python dictionary or JSON/YAML file.

Our goal is to make it easier to prototype, build, run, benchmark, optimize and manage complex AI/ML applications
across diverse and rapidly evolving models, data sets, software and hardware simply by chaining these
unified CM scripts into [portable, human-readable and reusable workflows](https://github.com/mlcommons/ck/blob/master/cm-mlops/script/app-image-classification-onnx-py/_cm.yaml).

Please go to this [GitHub page](https://github.com/mlcommons/ck)
to learn more about this community project
and join [public Discord server](https://discord.gg/JjWNWXKxwT)
to participate in collaborative developments.
Please check the [**Getting Started Guide**](https://github.com/mlcommons/ck/blob/master/docs/getting-started.md)
to understand how CM automation recipes work, how to use them to automate your own projects,
and how to implement and share new automations in your public or private projects.


### License
Expand Down
2 changes: 1 addition & 1 deletion cm/cmind/__init__.py
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
__version__ = "1.6.0.1"
__version__ = "1.6.1"

from cmind.core import access
from cmind.core import error
Expand Down

0 comments on commit c5ea901

Please sign in to comment.