How to use optimum-benchmark for custom testing of my model

[WCSY-YG]

I am currently using Intel® Extension for Transformers to quantize a model, and I wonder if it is possible to utilize optimum-benchmark for testing the model. Alternatively, if there are other methods to load large models, could I conduct tests using optimum-benchmark after loading the model? Many thanks; this has been a real challenge for me, as I'm unsure how to properly test an optimized large-scale model.

[IlyasMoutawwakil]

optimum-benchmark was built for cli usage, but using it as an api should be possible (I'm currently working on a refactorization for that). right now some features are easier to use than others (like trackers and generators).
but currently there's no direct way to pass a custom loaded model, it should be supported by one of the backends and its optimizations.

[IlyasMoutawwakil]

@WCSY-YG I just discovered that IPEX is being integrated directly in optimum-intel. If interested I would be glad to review a PR adding it as a backend in optimum-benchmark, thus enabling benchmarking it natively from the cli.

If it's not exactly what you're looking for, once #118 is merged, you'll be able to do things like creating your own custom backend by subclassing Backend and BackendConfig, for example, IPEXBackend and IPEXBackendConfig and then:

backend_config = IPEXBackendConfig(model="gpt2", no_weights=True)
launcher_config = ProcessConfig(device_isolation=True)
benchmark_config = InferenceConfig(memory=True)
experiment_config = ExperimentConfig(
    experiment_name="ipex-experiment",
    benchmark=benchmark_config,
    launcher=launcher_config,
    backend=backend_config,
)
report = launch(experiment_config)

You can also patch a backend with your custom model, for example:

backend_config = PyTorchConfig(model="gpt2", no_weights=True, device="cuda")
backend = PytorchBackend(backend_config)
backend.pretrained_model = your_custom_gpt2_like_model
benchmark.run(backend)

but this solution can't use the launchers feature, as the backend need to be created in an isolated process.

[WCSY-YG]

@WCSY-YG I just discovered that IPEX is being integrated directly in optimum-intel. If interested I would be glad to review a PR adding it as a backend in optimum-benchmark, thus enabling benchmarking it natively from the cli.

If it's not exactly what you're looking for, once #118 is merged, you'll be able to do things like creating your own custom backend by subclassing Backend and BackendConfig, for example, IPEXBackend and IPEXBackendConfig and then:

backend_config = IPEXBackendConfig(model="gpt2", no_weights=True)
launcher_config = ProcessConfig(device_isolation=True)
benchmark_config = InferenceConfig(memory=True)
experiment_config = ExperimentConfig(
    experiment_name="ipex-experiment",
    benchmark=benchmark_config,
    launcher=launcher_config,
    backend=backend_config,
)
report = launch(experiment_config)

You can also patch a backend with your custom model, for example:

backend_config = PyTorchConfig(model="gpt2", no_weights=True, device="cuda")
backend = PytorchBackend(backend_config)
backend.pretrained_model = your_custom_gpt2_like_model
benchmark.run(backend)

but this solution can't use the launchers feature, as the backend need to be created in an isolated process.

thankyou